4.1.1 Containing Nested Complexity

In JavaScript, deep nesting is one of the clearest signs of complexity. Understanding the code at any given nesting level involves understanding how the flow arrives there, the state at every level in scope, how the flow might break out of the level, and which other flows might lead to the same level. Granted, we don’t always need to keep all this derived information in our memory. The problem is that, when we do, we might have to spend quite a few minutes reading and understanding the code, deriving such information, and otherwise not fixing the bug or implementing the feature that we had set out to resolve in the first place.

Nesting is the underlying source of complexity in patterns such as “callback hell,” or “promise hell,” in which callbacks are nested on top of one another. The complexity has little to do with spacing, although when taken to the extreme, that does make code harder to read. Instead, the complexity exists at the seams, where we need to fully understand the context in order to go deep into the callback chain and make fixes or improvements. An insidious variant of callback hell is the one where we have logic in every nesting level. This variant is coincidentally the one we can observe most often in real applications: we rarely have callbacks as depicted in the following bit of code, partly because it’s immediately obvious that something is wrong. We should probably either change the API so that we get everything we need at once, or we could leverage a small library that takes care of the flow while eliminating the deep nesting we’d otherwise have in our own code:

getProducts(products => {
  getProductPrices(products, prices => {
    getProductDetails({ products, prices }, details => {
      // ...
    })
  })
})

When we have synchronous logic intermixed with asynchronous callbacks, things get more challenging. The problem here is, almost always, a coupling of concerns. When a program has a series of nested callbacks that also include logic in between, it can be a sign that we’re mixing flow-control concerns with business concerns. In other words, our program would be in a better place if we kept the flow separate from the business logic. By splitting the code that purely determines the flow from the rest, we can better isolate our logic into its individual components. The flow, in turn, also becomes clearer because it’s now spelled out in plain sight instead of interleaved with business concerns.

Suppose that each nesting level in a series of callbacks contains about 50 lines of code. Each function in the series needs to reference zero, one, or more variables in its parent scope. If it needs zero references to its immediate parent scope, we can safely move it up to the same scope as its parent. We can repeat this process until the function is at the highest possible level, given the variables it has to reference. When functions reference at least one variable from the parent scope, we could opt to leave them unchanged or to pass those references as parameters so that we can keep on decoupling the functions.

As we move logic into its own functions and flatten the callback chain, we’ll be left with the bare flow of operations being separate from the operations themselves. Libraries like contra can help manage the flow itself, while user code worries about business logic.

4.1.2 Feature Entanglement and Tight Coupling

As a module becomes larger, it also gets easier to mistakenly collapse distinct features together by interleaving their code in such a way that it is hard to reuse each feature independently, debug and maintain them, or otherwise extricate the features from one another.

For example, if we have a feature for notifying subscribers and a feature to send notifications, we could strive to keep the features apart by clearly defining how notifications can be constructed and handed off to a different service that then sends those notifications. That way, subscriber notifications can be sent through the notification service, but given the clear separation, we won’t be letting subscriber-specific notions get in the way of sending other kinds of notifications to our customers.

One way of reducing the risk of entanglement is to design features up front, being particularly on the lookout for concerns that could be componentized or otherwise clearly delineated. By doing a little work before sitting down to write code, we might avert the risks of tight coupling.

Being alert when reading old code can also be key in identifying what was previously a well-contained module that evolved to cover a broad range of concerns. We can then, over time, break these concerns into individual modules or better-isolated functions so that each concern is easier to maintain and understand separately.

Instead of trying to build a large feature all at once, we could build it from the inside out, keeping each stage of the process in functions that live at the same level instead of being deeply nested. Doing this methodically will lead to better decoupling, as we’ll move away from monolithic structures and toward a more modular approach, where functions have smaller scopes and take what they need in the form of parameters.

When we’d have to repeat ourselves by passing a lot of scope variables as function parameters just to avoid nested functions, a light degree of nesting is desirable to avoid this repetition. In key functional boundaries, where our concerns go from “gather model details” to “render HTML page” to “print HTML page to PDF,” nesting will invariably lead to coupling and less reusability, which is why repeating ourselves a little bit may be warranted in these cases.

4.1.3 Frameworks: The Good, the Bad, and the Ugly

Conventions are useful because they allow for better self-direction among developers, without causing lagoons of inconsistency to spread across our codebase. Chaos would ensue should we allow a team of developers too much freedom without sound design direction and conventions that dictate how different portions of an application should be shaped. A large number of conventions might hinder productivity, especially if some of our conventions appeared to work as if by magic.

When it comes to conventions, frameworks are a special case. Frameworks are packed to the brim with conventions and best practices. Some of them live in the library and tooling ecosystem around the framework, while many live in the shape our code takes when we rely on that framework. Upon adopting a framework, we’re buying into its conventions and practices. Most modern JavaScript frameworks offer ways of breaking our application into small chunks, regardless of whether the framework is for the client or server.

Express has middleware and routes; AngularJS has directives, services, and controllers; React has components; and so on and so forth. These conventions and abstractions are tremendously helpful to keep complexity in check while building an application. As our components grow larger, regardless of the abstraction or framework of choice, things will get more complicated. At this moment, we usually can refactor our code into smaller components that are then wrapped with larger ones, preserving separation of concerns, and keeping complexity on a short leash.

Eventually, we’ll come across requirements that don’t exactly fit the mold proposed by our framework of choice. Generally, this means that the required functionality belongs on a separate layer. For example, Express in Node.js is a framework concerned with handling HTTP requests and serving responses. If one of our API endpoints needs to result in an email being sent, we could embed email-sending logic in the controller for that API endpoint. However, if an API endpoint controller is already concerned with, say, publishing blog posts, then it would be hardly right to embed email-sending logic on that same controller since it’s a different concern entirely. Instead, what we could do is create a subscribers service component, with functionality such as subscribe, which adds a subscriber after verifying their email, and notify, which takes care of sending the emails. Taking this idea further still, perhaps most of the work in subscribers.notify should occur via yet another service component called emails, which takes care of properly configuring our email-sending capability, and also has functionality to turn would-be emails into plain console.log statements for quick access to the contents of the emails during debug sessions.

Having clearly defined layers is paramount to the design of effective and maintainable applications once we’re past the prototyping stages. Layers can be made up of components that follow the conventions proposed by the frameworks we use, or they can be self-imposed, like the service layer discussed in the previous paragraph. Using layers, and as long as we favor function parameters over scope for context passing, we can introduce horizonal scaling by placing several orthogonal components alongside each other, without letting them run into each other’s concerns.

4.2.1 Embracing Variables over Clever Code

Complex code is predominantly shorter than it should be, and often deceitfully so. An expression that might have involved 5 to 10 short lines of code usually ends up being represented in 1 or 2 clever lines of code. The problem with clever code is that we need to expend time and energy to read it whenever its intent is not clear in our mind, which is only the case when we first write the code or right after spending considerable time analyzing it.

One of the underlying issues that can be identified when reading complex code is that it uses few variables. In the dawn of programming, memory resources were scarce, so programmers had to optimize allocation, and this often meant reusing variables and using fewer of them. In modern systems, we don’t need to treat memory as a sacred, precious, and limited resource. Instead, we can focus on making programs readable to both our future selves and fellow developers.

Readability is better served by an abundance of properly named variables or functions than by sparsity. Consider the following example, part of a larger routine; a program ensures that the user is currently logged in with a valid session, and otherwise redirects the user to a login page:

if (
  auth !== undefined &&
  auth.token !== undefined &&
  auth.expires > Date.now()
) {
  // we have a valid token that hasn't expired yet
  return
}

As the routine becomes larger, we collect if statements with nonobvious or complicated clauses, such as the reason we’re checking that auth has a token value if we’re not doing anything with it here. The solution is usually to add a comment explaining the reason this check exists. In this case, the comment tells us this is a valid token that hasn’t expired. We could turn that comment into code, and simplify the if statement in the process, by creating a small function that breaks down the conditional, as shown next:

function hasValidToken(auth) {
  if (auth === undefined || auth.token === undefined) {
    return false
  }
  const hasNotExpiredYet = auth.expires > Date.now()
  return hasNotExpiredYet
}

We can now turn our if statement plus comment into a function call, as shown in the following bit of code. Certainly, the totality of our refactored code is a bit longer, but now it’s self-descriptive. Code that describes what it does in the process of doing it doesn’t require as many comments, and that’s important because comments can become easily outdated. Moreover, we’ve extracted the long conditional in the if statement to a function, which keeps us more focused while parsing the codebase. If every condition or task was inline, we’d have to understand everything in order to understand how a program works. When we offload tasks and conditions to other functions, we’re letting readers know they can have faith that hasValidToken will check for validity of the auth object, and the conditional becomes a lot easier to digest:

if (hasValidToken(auth)) {
  return
}

We could’ve used more variables without creating a function, inlining the computation of hasValidToken right before the if check. A crucial difference between the function-based refactor and the inlining solution is that we used a short-circuiting return statement to preemptively bail when we already knew the token was invalid.¹ However, we can’t use return statements to bail from the snippet that computes hasValidToken in the following piece of code without coupling its computation to knowledge about what the routine should return for failure cases. As a result, our only options are tightly coupling the inline subroutine to its containing function, or using a logical or ternary operator in the intermediate steps of the inlined computation:

const hasToken = auth === undefined || auth.token === undefined
const hasValidToken = hasToken && auth.expires > Date.now()
if (hasValidToken) {
  return
}

Both of these options have their downsides. If we couple the return statements with the parent function, we’ll need to be careful if we want to replicate the logic elsewhere, as the return statements and possibly their logic will have to adapt as well. If we decide to use ternary operators as a way of short-circuiting, we’ll end up with logic that might be as complex as the code we originally had in the if statement.

Using a function not only avoids these two problems, thanks to the ability to return intermediate results, but also defers reasoning about its contents until we actually need to understand how tokens are checked for validity.

Although moving conditionals to a function might sound like a trivial task, this approach is at the heart of modular design. It is by composing small bits of complexity using several additive functions that we can build large applications that are less straining to read. A large pool of mostly trivial functions can add up to a veritable codebase in which each bit of code is relatively isolated and easy to understand, provided we trust that functions do what their names say they do. In this vein, it is of utmost importance to think long and deep about the name of every function, every variable, and every package, directory, or data structure we conceive.

When used deliberately and extensively, early returns—sometimes referred to as guard clauses or short-circuits—can be unparalleled when it comes to making an application as readable as possible. Let’s explore this concept in further detail.

4.2.2 Guard Clauses and Branch Flipping

When we have a long branch inside a conditional statement, chances are we’re doing something wrong. Pieces of code like the following are commonplace in real-world applications, with a long success case branch taking up significant amounts of code while having several else branches sprinkled near the end that would log an error, throw, return, or otherwise perform a failure-handling action:

if (response) {
  if (!response.errors) {
    // ... use `response`
  } else {
    return false
  }
} else {
  return false
}

In the example, we’re optimizing readability for the success case, while the failure handling is relegated to the very end of our piece of code. There are several problems with this approach. For one, we have to indulge in unnecessary nesting of every success condition, or otherwise put them all in a huge conditional statement. Although it’s rather easy to understand the success case, things can get tricky when we’re trying to debug programs like this, because we need to keep the conditionals in our head the whole time we’re reading the program.

A better alternative is to flip the conditionals, placing all failure-handling statements near the top. Though counterintuitive at first, this approach has several benefits. It reduces nesting and eliminates else branches, while promoting failure handling to the top of our code. This has the added benefit that we’ll become more aware of error handling and naturally gravitate toward thinking about the failure cases first. This is a great trait to have when doing application development, where forgetting to handle a failure case might result in an inconsistent experience for end users with a hard-to-trace error on top. The following example illustrates the early exit approach:

if (!response) {
  return false
}
if (response.errors) {
  return false
}
// ... use `response`

As stated previously, this early-exit approach is often referred to as guard clauses. One of their biggest benefits is that we can learn all the failure cases upon reading the first few lines of a function or piece of code. We’re not limited to return statements; we could throw errors in a promise-based context or in an async function, and in callback chaining contexts we might opt for a done(error) callback followed by a return statement.

Another benefit of guard clauses is almost implicit: given that they’re placed near the top of a function, we have quick access to its parameters, we can better understand how the function validates its inputs, and we can more effectively decide whether we need to add new guard clauses to improve validation rules.

Guard clauses don’t tell the reader everything they need to know that might go wrong when calling a function, but they provide a peek into expected immediate failure cases. Other things that might go wrong lie in the implementation details of the function. Perhaps we use a different service or library to fulfill the bulk of our function’s task, and that service or library comes with its own set of nested guard clauses and potential failure cases that will bubble up all the way to our own function’s outcome.

4.2.3 An Interdependency Pyramid

Writing straightforward code is not all that different from writing other straightforward text. Text is often arranged in paragraphs, which are somewhat comparable with functions; we can consider their input to be the reader’s knowledge and everything else they’ve read so far in the text, and the output to be what the reader gets out of the paragraph.

Within a book chapter or any other piece of long-form text, paragraphs are organized in a sequential manner, allowing the reader time to digest each paragraph before jumping onto the next. The logical sequence is very much intentional: without a coherent sequencing, it would be nearly impossible to make sense of a text. Thus, writers optimize for making sure concepts are introduced before they’re discussed, providing context to the reader.

Function expressions such as the one in the next snippet won’t be assigned to the variable binding until the line containing the assignment is evaluated. Until then, the variable binding exists in the scope, thanks to hoisting, but it is undefined until the assignment statement is evaluated:

double(6) // TypeError: double is not a function
var double = function(x) {
  return x * 2
}

Furthermore, if we’re dealing with a let or const binding, then TDZ semantics produce an error if we reference the binding at all before the variable declaration statement is reached:

double(6) // TypeError: double is not defined
const double = function(x) {
  return x * 2
}

Function declarations like the one in the following snippet, in contrast, are hoisted to the top of the scope. This means we can reference them anywhere in our code:

double(6) // 12
function double(x) {
  return x * 2
}

Now, I mentioned that text is written sequentially, and that writers avoid surprises by presenting concepts before discussing them. Establishing a context in a program is a different endeavor, however. If we have a module that has the goal of rendering a chart with user engagement statistics, the top of the function should address things the reader already knows—namely, the high-level flow for what the rendering function is meant to do: analyze the data, construct some data views, and model that data into something we can feed into a visualization library that then renders the desired chart.

What we have to avoid is jumping directly into unimportant functions such as a data-point label formatter, or the specifics of the data modeling. By keeping only the high-level flow near the top, and the specifics toward the end, complex functionality can be designed in such a way that readers experience a zoomed-out overview of the functionality at first, and as they read the code, they uncover the details of the way this chart was implemented.

In a concrete sense, this means we should present functions in a codebase in the order that they’ll be read by the consumer (a first-in, first-out queue), and not in the execution order (a last-in, first-out stack). Computers do as they’re told and dig ever deeper into the flow, executing the most deeply nested routines before jumping out of a series of subroutines and executing the next line. But this is an unfruitful way for humans to read a codebase, given we’re ill-suited to keeping all that state in our heads.

Perhaps a more specific analogy for this kind of spiraling approach can be found in newspaper articles; the author typically offers a title that describes an event at the highest possible level, and then follows up with a lead paragraph that summarizes what happened, again at a high level. The body of the article starts also at a high level, carefully avoiding to spook the reader with too many details. It is only midway through the article that we’ll start finding details about the event that, aided by the context set forth at the beginning of the article, can give us a complete picture of what transpired.

Given the stack-based nature of programming, it’s not that easy to naturally approach programs as if they were newspaper articles. We can, however, defer execution of implementation details to other functions or subroutines, and thanks to hoisting, we can place those subroutines after their higher-level counterparts. In doing so, we’re organizing our programs in a way that invites readers in, shows them a few high-level hints, and then gradually unveils the spooky details of the way a feature is implemented.

4.2.4 Extracting Functions

Deliberate, pyramidal structures that deal with higher-level concerns near the top and switch to more-specific problems as we go deeper into the inner workings of a system work wonders in keeping complexity on a tight leash. Such structures are particularly powerful because they break complex items into their own individual units near the flat bottom of the system, avoiding a complicated interweaving of concerns that are fuzzied together, becoming indistinguishable from one another over time.

Pushing anything that gets in the way of the current flow to the bottom of a function is an effective way of streamlining readability. As an example, imagine that we have a nontrivial mapper inline, in the heart of a function. In the following code snippet, we’re mapping the users into user models, as we often need to do when preparing JSON responses for API calls:

function getUserModels(done) {
  findUsers((err, users) => {
    if (err) {
      done(err)
      return
    }

    const models = users.map(user => {
      const { name, email } = user
      const model = { name, email }
      if (user.type.includes('admin')) {
        model.admin = true
      }
      return model
    })

    done(null, models)
  })
}

Now compare that to the following bit of code, where we extract the mapping function and shove it out of the way. Given that the mapping function doesn’t need any of the scope from getUserModels, we can pull it out of that scope entirely, without needing to place toUserModel at the bottom of the getUserModels function. This means we can now also reuse toUserModel in other routines. We don’t have to wonder whether the function actually depends on any of the containing scope’s context anymore, and getUserModels is now focused on the higher-level flow where we find users, map them to their models, and return them:

function getUserModels(done) {
  findUsers((err, users) => {
    if (err) {
      done(err)
      return
    }

    const models = users.map(toUserModel)

    done(null, models)
  })
}

function toUserModel(user) {
  const { name, email } = user
  const model = { name, email }
  if (user.type.includes('admin')) {
    model.admin = true
  }
  return model
}

Furthermore, if additional work needed to be done between the mapping and the callback, that work could also be moved into another small function that wouldn’t get in the way of our higher-level getUserModels function.

A similar case occurs when we have a variable that’s defined based on a condition, as shown in the next snippet. Bits of code like this can distract the reader away from the core purpose of a function, to the point where it’s often ignored or glossed over:

// ...
let website = null
if (user.details) {
  website = user.details.website
} else if (user.website) {
  website = user.website
}
// ...

It’s best to refactor this kind of assignment into a function, like the one shown next. Note that we include a user parameter so that we can push the function out of the scope chain where we’ve originally defined the user object, and at the same time go from a let binding to a const binding. When reading this piece of code later down the line, the benefit of const is that we’ll know the binding won’t change. With let, we can’t be certain that bindings won’t change over time, adding to the pile of things the reader should be watching out for when trying to understand the algorithm:

// ...
const website = getUserWebsite(user)
// ...

function getUserWebsite(user) {
  if (user.details) {
    return user.details.website
  }
  if (user.website) {
    return user.website
  }
  return null
}

Regardless of your flavor of choice when it comes to variable binding, bits of code that select a slice of application state are best shoved away from the relevant logic that will use this selected state to perform an action. This way, we’re not distracted by concerns about how state is selected, instead of focusing on the action that our application logic is trying to carry out.

When we want to name an aspect of a routine without adding a comment, we could create a function to host that functionality. Doing so not only gives a name to what the algorithm is doing, but also allows us to push that code out of the way, leaving behind only the high-level description of what’s going to happen.

4.2.5 Flattening Nested Callbacks

Codebases with asynchronous code flows often fall into callback hell; each callback creates a new level of indentation, making code harder and harder to read as we approach the deep end of the asynchronous flow chain:

a(function () {
  b(function () {
    c(function () {
      d(function () {
        console.log('hi!')
      })
    })
  })
})

The foremost problem with this kind of structure is scope inheritance. In the deepest callback, passed to the g function, we’ve inherited the combined scopes of all the parent callbacks. As functions become larger, and more variables are bound into each of these scopes, it becomes ever more challenging to understand one of the callbacks in isolation from its parents.

This kind of coupling can be reverted by naming the callbacks and placing them all in the same nesting level. Named functions may be reused in other parts of our component, or exported to be used elsewhere. In the following example, we’ve eliminated up to three levels of unnecessary nesting, and by eliminating nesting, we’ve made the scope for each function more explicit:

a(a1)
function a1() {
  b(b1)
}
function b1() {
  c(c1)
}
function c1() {
  d(d1)
}
function d1() {
  console.log('hi!')
}

When we do need some of the variables that existed in the parent scope, we can explicitly pass them on to the next callback in the chain. The following example passes an arrow function to d, as opposed to passing the d1 callback directly. When executed, the arrow function ends up calling d1 anyway, but now it has the additional parameters we needed. These parameters can come from anywhere, and we can do this throughout the chain, while keeping it all in the same indentation level:

a(a1)
function a1() {
  b(b1)
}
function b1() {
  c(c1)
}
function c1() {
  d(() => d1('hi!'))
}
function d1(salute) {
  console.log(salute) // <- 'hi!'
}

Now, this could also be resolved using a library such as async, which simplifies the flattened chaining process by establishing patterns. The async.series method accepts an array of task functions. When called, the first task is executed, and async waits until the next callback is invoked before jumping onto the next task. When all tasks have been executed, or an error arises in one of the tasks, the completion callback in the second argument passed to async.series is executed. In the following illustrative example, each of the three tasks is executed in series, one at a time, waiting a second before each task signals its own completion. Lastly, the 'done!' message is printed to the console:

async.series([
  next => setTimeout(() => next(), 1000),
  next => setTimeout(() => next(), 1000),
  next => setTimeout(() => next(), 1000)
], err => console.log(err ? 'failed!' : 'done!'))

Libraries like async come with several ways of mixing and matching asynchronous code flows, in series or concurrent, allowing us to pass variables between callbacks without having to nest together entire asynchronous flows.

Naturally, callbacks aren’t the only asynchronous flow pattern that might end up in hell. Promises can end up in this state just as easily, as shown in this contrived snippet:

Promise.resolve(1).then(() =>
  Promise.resolve(2).then(() =>
    Promise.resolve(3).then(() =>
      Promise.resolve(4).then(value => {
        console.log(value) // <- 4
      })
    )
  )
)

A similar piece of code that wouldn’t be affected by the nesting problem is shown next. Here, we’re taking advantage of promises behaving in a tree-like manner. We don’t necessarily need to attach reactions onto the last promise, and instead, we can return those promises so that the chaining can always occur at the top level, allowing us to avoid any and all scope inheritance:

Promise.resolve(1)
  .then(() => Promise.resolve(2))
  .then(() => Promise.resolve(3))
  .then(() => Promise.resolve(4))
  .then(value => {
    console.log(value) // <- 4
  })

Similarly, using async functions can turn what was previously a promise-based flow and turn it into something that can be mapped to our own mental model of the program’s execution flow. The following bit of code is similar to the preceding snippet, but uses async/await instead:

async function main() {
  await Promise.resolve(1)
  await Promise.resolve(2)
  await Promise.resolve(3)
  const value = await Promise.resolve(4)
  console.log(value) // <- 4
}

4.2.6 Factoring Similar Tasks

We’ve already discussed at length why creating abstractions isn’t always the best way of reducing complexity in an application. Abstractions can be particularly damaging when created too early: at the time, we might not have enough information about the shape and requirements for other components that we might want to hide behind the abstraction layer. Over time, we might end up aggressively shaping components only so that they fit the abstraction, which could have been avoided by not settling for an abstraction too early.

When we do avoid creating abstractions prematurely, we’ll start noticing functions that have an uncanny resemblance to the shape of similar functions. Maybe the flow is identical, maybe the output is similar, or maybe all that really changes is we’re accessing an attribute named href in one case and an attribute named src in another case.

Consider the case of an HTML crawler that needs to pull out snippets of an HTML page and reuse them later in a different context. Among other things, this crawler needs to take relative resource locators like /weekly and resolve them to absolute endpoints like https://ponyfoo.com/weekly, depending on the origin of the resource. This way, the HTML snippets can then be repurposed on other mediums, such as on a different origin or a PDF file, without breaking the end-user experience.

The following code takes a piece of HTML and transforms a[href] and img[src] into absolute endpoints by using the $ jQuery-like DOM utility library:

function absolutizeHtml(html, origin) {
  const $dom = $(html)
  $dom.find('a[href]').each(function () {
    const $element = $(this)
    const href = $element.attr('href')
    const absolute = absolutize(href, origin)
    $element.attr('href', absolute)
  })
  $dom.find('img[src]').each(function () {
    const $element = $(this)
    const src = $element.attr('src')
    const absolute = absolutize(src, origin)
    $element.attr('src', absolute)
  })
  return $dom.html()
}

Because the function is small, it’d be perfectly acceptable to keep absolutizeHtml as is. However, if we later decide to add iframe[src], script[src], and link[href] to the list of attributes that might contain endpoints we want to transform, we’ll probably want to avoid having five copies of the same routine. That’s more likely to be confusing and result in changes being made to one of them without being mirrored in the other cases, increasing complexity.

The following bit of code keeps all attributes we want to transform in an array, and abstracts the repeated bit of code so that it’s reused for every tag and attribute:

const attributes = [
  ['a', 'href'],
  ['img', 'src'],
  ['iframe', 'src'],
  ['script', 'src'],
  ['link', 'href']
]

function absolutizeHtml(html, origin) {
  const $dom = $(html)
  attributes.forEach(absolutizeAttribute)
  return $dom.html()

  function absolutizeAttribute([ tag, property ]) {
    $dom.find(`${ tag }[${ property }]`).each(function () {
      const $element = $(this)
      const value = $element.attr(property)
      const absolute = absolutize(value, origin)
      $element.attr(property, absolute)
    })
  }
}

A similar situation occurs when we have a concurrent flow that remains more or less constant across multiple functions. In this case, we might want to consider keeping the flow in its own function, and passing a callback for the actual processing logic that is different in each case.

In other cases, we might notice that a few different components all need the same piece of functionality. Commenting features often fall into this case, where different components such as user profiles, projects, or artifacts might need the ability to receive, show, edit, and delete comments. This case can be interesting because the business requirement is not always identified up front, and we might embed the child feature into the parent component before realizing it’d be useful to extract the feature so that it can be reused in other parent components. While this sounds obvious in hindsight, it’s not always clear when we’ll need to reuse functionality somewhere else. Keeping every aspect of functionality isolated just in case we need to reuse any can be costly in terms of time and development effort.

More often than not, however, abstractions can end up complicating matters. The trade-off might not be worth it because the code becomes much harder to read, or the underlying code might not be mature enough yet. Maybe we don’t know what special requirements we may end up with for other objects adopting similar functionality, meaning we’re not comfortable creating an abstraction that could lead to unforeseen problems in the future.

Whenever we are uncertain about whether an abstraction is up to muster, it pays to go back to the original piece of code we had before introducing the abstraction, and comparing the two pieces. Is the new piece easier to understand, modify, and consume? Would that still be the case as a newcomer? Try to consider how the outcome of those questions would change if you hadn’t looked at this code in a while. Ask your coworkers for their opinion, too; because they haven’t seen that code yet and may end up having to consume it, they’re great candidates to help decide which approach is better.

4.2.7 Slicing Large Functions

Consider breaking what would otherwise inevitably be a single large function into smaller functions. These may be organized by splitting functionality by steps or by each aspect of the same task. All of these functions should still always rely on guard clauses to do all of our error checking up front, ensuring that state is constrained by what we allow it to be at each point in time.

The overall structure of your typical function should begin with guard clauses, making sure the input we receive is what we expect: enforcing required parameters, their correct data types, correct data ranges, and so on. If these inputs are malformed, we should bail immediately. This ensures that we don’t work with inputs we’re unprepared to deal with, and ensures that the consumers get an error message explaining the root reason for not getting the results they expect (as opposed to a message that might involve debugging work), such as undefined is not a function caused by trying to call an input that was supposed to be a function but wasn’t, or was supposed to result in our routine finding a function, but didn’t.

Once we know the inputs are well formed, data processing can begin. We’ll transform the inputs, map them to the output we want to produce, and return that output. Here we have the opportunity to break the function into several pieces. Each aspect of the transformation of inputs into output is potentially its own function. The way of reducing complexity in a function is not by collapsing hundreds of lines of code into tens of complicated lines of code. Instead, we can move each of these long pieces of code into individual functions that deal with only one aspect of the data. Those functions can then also be hoisted out of our function and onto its parent scope, showing that there wasn’t a reason that a particular aspect of transforming the inputs had to be coupled to the entire function doing the transformation.

Each aspect of a transformation operation can be analyzed and moved into its own function. The smaller function may take a few of the inputs in the larger function, or perhaps some of the intermediate values that were produced in the larger function. It can then conduct its own input sanitization, and be broken apart even further. The process of identifying aspects of an operation that can be recursively compartmentalized and moved into their own functions is highly effective because it allows for dauntingly large functions to be broken into simpler pieces that aren’t as daunting to refactor.

At first, we can identify the three or four largest aspects of a function, and break those apart. The first part might involve filtering out the parts of the input we’re not interested in, the second might involve mapping that into something else, and the third part might involve merging all of the data together. Once we’ve identified each aspect of the function, we might break those into their own functions, with their own inputs and output. Subsequently, we can do this for each of those smaller functions.

We can keep doing this for as long as there’s the opportunity for the functions to be simplified. As discussed in the previous section, it’s valuable to take a step back after each of these refactors, and evaluate whether the end result is indeed simpler and easier to work with than what we had before it was refactored.

4.3.1 Current State: It’s Complicated

The problem with state is that, as an application grows, its state tree inevitably grows with it, and for this reason large applications are hopelessly complex. This complexity exists in the whole, but not necessarily in individual pieces. This is why breaking an application into ever smaller components might reduce local complexity even when it increases overall complexity. That is to say, breaking a single large function into a dozen small functions might make the overall application more complex, as there would be 10 times as many pieces. But it also makes the individual aspects of the previously large function that are now covered by each small function simpler when we’re focused on them, as thus easier-to-maintain individual pieces of a large, complicated system, without requiring a complete or even vast understanding of the system as a whole.

At its heart, state is mutable. Even if the variable bindings themselves are immutable, the complete picture is mutable. A function might return a different object every time, and we may even make that object immutable so that the object itself doesn’t change either, but anything that consumes the function receives a different object each time. Different objects mean different references, meaning the state as a whole mutates.

Consider a game of chess: each of two players starts with 16 pieces, each deterministically assigned a position on a checkerboard. The initial state is always the same. As each player inputs their actions, moving and trading pieces, the system state mutates. A few moves into the game, there is a good chance we’ll be facing a game state we haven’t ever experienced before. Computer program state is a lot like a game of chess, except there’s more nuance in the way of user input, and an infinitude of possible board positions and state permutations.

In the world of web development, a human decides to open a new tab in their favorite web browser and then search Google for “cat in a pickle gifs.” The browser allocates a new process through a system call to the operating system, which shifts some bits around on the physical hardware that lies inside the human’s computer. Before the HTTP request hits the network, we need to hit DNS servers, engaging in the elaborate process of casting google.com into an IP address. The browser then checks whether a ServiceWorker is installed, and assuming there isn’t one, the request finally takes the default route of querying Google’s servers for the phrase “cat in a pickle gifs.”

Naturally, Google receives this request at one of the frontend edges of its public network, in charge of balancing the load and routing requests to healthy backend services. The query goes through a variety of analyzers that attempt to break it down to its semantic roots, stripping the query down to its essential keywords to better match relevant results.

The search engine figures out the 10 most relevant results for “cat pickle gif” out of billions of pages in its index, which was of course primed by a different system that’s also part of the whole. At the same time, Google pulls down a highly targeted piece of relevant advertisement about cat gifs that matches what it believes is the demographic the human making the query belongs to, thanks to a sophisticated ad network that figures out whether the user is authenticated with Google through an HTTP header session cookie. The search results page starts being constructed and streamed to the human, who now appears impatient and fidgety.

As the first few bits of HTML begin streaming down the wire, the search engine produces its result and hands it back to the frontend servers, which include it in the HTML stream that’s sent back to the human. The web browser has been working hard at this too, parsing the incomplete pieces of HTML that have been streaming down the wire as best it could, even daring to launch other admirably and equally mind-boggling requests for HTTP resources presumed to be JavaScript, CSS, font, and image files as the HTML continues to stream down the wire. The first few chunks of HTML are converted into a DOM tree, and the browser would finally be able to begin rendering bits and pieces of the page on the screen, if it weren’t for the pending, equally mind-boggling CSS and font requests.

As the CSS stylesheets and fonts are transmitted, the browser begins modeling the CSS Object Model (CSSOM) and getting a more complete picture of how to turn the HTML and CSS plain-text chunks provided by Google servers into a graphical representation that the human finds pleasant. Browser extensions get a chance to meddle with the content, removing the highly targeted piece of relevant advertisement about cat gifs before I even realize Google hoped I wouldn’t block ads this time around.

A few seconds have passed since I first decided to search for cat in a pickle gifs. Needless to say, thousands of others brought similarly inane requests. To the same systems. During this time.

Not only does this example demonstrate the marvelous machinery and infrastructure that fuels even our most flippant daily computing experiences, but it also illustrates how abundantly hopeless it is to make sense of a system as a whole, let alone its comprehensive state at any given point in time. After all, where do we draw the boundaries? Within the code we wrote? The code that powers our customers’ computers? Their hardware? The code that powers our servers? Its hardware? The internet as a whole? The power grid?

4.3.2 Eliminating Incidental State

We’ve established that the overall state of a system has little to do with our ability to comprehend parts of that same system. Our focus on reducing state-based entropy must then lie in the individual aspects of the system. It’s for this reason that breaking apart large pieces of code is so effective. We’re reducing the amount of state local to each given aspect of the system, and that’s the kind of state that’s worth taking care of, since it’s what we can keep in our heads and make sense of.

Whenever persistence is involved, a discrepancy is going to exist between ephemeral state and realized state. In the case of a web application, we could define ephemeral state as any user input that hasn’t resulted in state being persisted yet, as might be the case of an unsaved user preference that might be lost unless persisted. We can say realized state is the state that has been persisted, and that different programs might have different strategies on how to convert ephemeral state into realized state. A web application might adopt an offline-first pattern in which ephemeral state is automatically synchronized to an IndexedDB database in the browser, and eventually realized by updating the state persisted on a backend system. When the offline-first page is reloaded, unrealized state may be pushed to the backend or discarded.

Incidental state can occur when we have a piece of data that’s used in several parts of an application, and that is derived from other pieces of data. When the original piece of data is updated, it wouldn’t be hard to inadvertently leave the derived pieces of data in their current state, making them stale when compared to the updated original pieces of data. As an example, consider a piece of user input in Markdown and the HTML representation derived from that piece of Markdown. If the piece of Markdown is updated but the previously compiled pieces of HTML are not, then different parts of the system might display different bits of HTML out of what was apparently the same single Markdown source.

When we persist derived state, we’re putting the original and the derived data at risk of falling out of sync. This isn’t the case just when dealing with persistence layers, but can also occur in a few other scenarios as well. When dealing with caching layers, their content may become stale because the underlying original piece of content is updated but we forget to invalidate pieces of content derived from the updated data. Database denormalization is another common occurrence of this problem, whereby creating derived state can result in synchronization problems and stale byproducts of the original data.

This lack of synchronization is often observed in discussion forum software, as user profiles are denormalized into comment objects in an effort to save a database roundtrip. When users later update their profile, however, their old comments preserve a stale avatar, signature, or display name. To avoid this kind of issue, we should always consider recomputing derived state from its roots. Even though doing so won’t always be possible, performant, or even practical, encouraging this kind of thinking across a development team will, if anything, increase awareness about the subtle intricacies of denormalized state.

As long as we’re aware of the risks of data denormalization, we can then indulge in it. A parallel could be drawn to the case of performance optimization, as we should be aware that attempting to optimize a program based on microbenchmarks instead of data-driven optimization will most likely result in wasted developer time. Furthermore, just as with caches and other intermediate representations of data, performance optimization can lead to bugs and code that’s ultimately harder to maintain. This is why neither should be embarked upon lightly, unless in a certain business case where performance is hurting the bottom line.

4.3.3 Containing State

State is inevitable. As we discussed in Section 4.3.1: Current State: It’s Complicated, the full picture hardly affects our ability to maintain small parts of that state tree. In the local case—each of the interrelated but ultimately separate pieces of code we work with day to day—all that matters are the inputs we receive and the outputs we produce. That said, generating a large amount of output when we could instead emit a single piece of information is undesirable.

When all intermediate state is contained inside a component instead of being leaked to others, we’re reducing the friction in interacting with our component or function. The more we condense state into its smallest possible representation for output purposes, the better contained our functions will become. Incidentally, we’re making the interface easier to consume. Since there’s less state to draw from, there are fewer ways of consuming that state. This reduces the number of possible use cases, but by favoring composability over serving every possible need, we’re making each piece of functionality, when evaluated on its own, simpler.

One other case in which we may incidentally increase complexity occurs whenever we modify the property values of an input. This type of operation should be made extremely explicit, so as to not be confused, and avoided where possible. If we assume functions to be defined as the equation between the inputs we receive and the outputs we produce, the side effects are ill-advised. Mutations on the input within the body of a function are one example of side effects, which can be a source of bugs and confusion, particularly due to the difficulty in tracking down the source for these mutations.

It is common to observe functions that modify an input parameter and then return that parameter. This is often the case with Array#map callbacks, where the developer wants to change a property or two on each object in a list, but also to preserve the original objects as the elements in the collection, as shown in the following example:

movies.map(movie => {
  movie.profit = movie.gross - movie.budget
  return movie
})

In these cases, it might be best to avoid using Array#map altogether, using Array#forEach or for..of instead, as shown here:

for (const movie of movies) {
  movie.profit = movie.gross - movie.budget
}

Neither Array#forEach nor for..of allow for chaining, assuming you wanted to filter the movies by criteria such as “profit is greater than $15M”; they’re pure loops that don’t produce any output. This is a good problem to have, however, because it explicitly separates data mutations at the movie item level, where we’re adding a profit property to each item in movies, from transformations at the movies level, where we want to produce an entirely new collection consisting of only expensive movies:

for (const movie of movies) {
  movie.profit = movie.amount * movie.unitCost
}
const successfulMovies = movies.filter(
  movie => movie.profit > 15
)

Relying on immutability is an alternative that doesn’t involve pure loops nor resort to breakage-prone side effects.

4.3.4 Leveraging Immutability

The following example takes advantage of the object spread operator to copy every property of movie into a new object, and then adds a profit property to it. Here we’re creating a new collection, made up of new movie objects:

const movieModels = movies.map(movie => ({
  ...movie,
  profit: movie.amount * movie.unitCost
}))
const successfulMovies = movieModels.filter(
  movie => movie.profit > 15
)

Thanks to us making fresh copies of the objects we’re working with, we’ve preserved the movies collection. If we assume at this point that movies was an input to our function, we could say that modifying any movie in that collection would’ve made our function impure, since it’d have the side effect of unexpectedly altering the input.

By introducing immutability, we’ve kept the function pure. That means that its output depends on only its inputs and that we don’t create any side effects such as changing the inputs themselves. This, in turn, guarantees that the function is idempotent; calling a function repeatedly with the same input always produces the same result, given the output depends solely on the inputs and there are no side effects. In contrast, the idempotence property would’ve been brought into question if we had tainted the input by adding a profit field to every movie.

Large amounts of intermediate state or logic that permutates data into different shapes, back and forth, may be a signal that we’ve chosen poor representations of our data. When the right data structures are identified, we’ll notice that a lot less transformation, mapping, and looping are involved in getting inputs to become the outputs we need to produce. In the next section we’ll dive deeper into data structures.

4.4.1 Isolating Data and Logic

Keeping data strictly separate from methods that modify or access those data structures can help reduce complexity. When data is not cluttered with functionality, it becomes detached from it and thus easier to read, understand, and serialize. At the same time, the logic that was previously tied to our data can now be used when accessing different bits of data that share some trait with it.

As an example, the following piece of code shows a piece of data that’s encumbered by the logic that works with it. Whenever we want to leverage the methods of Value, we’ll have to box our input in this class, and if we later want to unbox the output, we’ll need to cast it with a custom-built valueOf method or similar:

class Value {
  constructor(value) {
    this.state = value
  }
  add(value) {
    this.state += value
    return this
  }
  multiply(value) {
    this.state *= value
    return this
  }
  valueOf() {
    return this.state
  }
}
console.log(+new Value(5).add(3).multiply(2)) // <- 16

Consider now, in contrast, the following piece of code. Here we have a couple of functions that purely compute addition and multiplication of their inputs, which are idempotent, and which can be used without boxing inputs into instances of Value, making the code more transparent to the reader. The idempotence aspect is of great benefit, because it makes the code more digestible: whenever we add 3 to 5, we know the output will be 8, whereas whenever we add 3 to the current state, we know only that Value will increment its state by 3:

function add(current, value) {
  return current + value
}
function multiply(current, value) {
  return current * value
}
console.log(multiply(add(5, 3), 2)) // <- 16

Taking this concept beyond basic mathematics, we can begin to see how this decoupling of form and function, or state and logic, can be increasingly beneficial. It’s easier to serialize plain data over the wire, keep it consistent across different environments, and make it interoperable regardless of the logic, than if we tightly coupled data and the logic around it.

Functions are, to a certain degree, hopelessly coupled to the data they receive as inputs: in order for the function to work as expected, the data it receives must satisfy its contract for that piece of input. Within the bounds of a function’s proper execution, the data must have a certain shape, traits, or adhere to whatever restrictions the function has in place. These restrictions may be somewhat lax (e.g., “must have a toString method”), highly specific (e.g., “must be a function that accepts three arguments and returns a decimal number between 0 and 1”), or anywhere in between. A simple interface is usually highly restrictive (e.g., accepting only a Boolean value). Meanwhile, it’s common for loose interfaces to become burdened by their own flexibility, leading to complex implementations that attempt to accommodate many shapes and sizes of the same input parameter.

We should aim to keep logic restrictive and only as flexible as deemed necessary by business requirements. When an interface starts out being restrictive, we can always slowly open it up later as new use cases and requirements arise. By starting out with a small use case, we’re able to grow the interface into something that’s naturally better fit to handle specific, real-world use cases.

Data, on the other hand, should be transformed to fit elegant interfaces, rather than trying to fit the same data structure into every function. Doing so would result in frustration similar to that caused by a rushed abstraction layer that doesn’t lend itself to being effortlessly consumed to leverage the implementations underlying it. These transformations should be kept separate from the data itself, so as to ensure reusability of each intermediate representation of the data on its own.

4.4.2 Restricting and Clustering Logic

Should a data structure—or code that leverages that data structure—require changes, the ripple effects can be devastating when the relevant logic is sprinkled all across the codebase. Consequently, when this happens, we need to update code from all over, making a point of not missing any occurrences, updating and fixing test cases as we go, and testing some more to certify that the updates haven’t broken down our application logic, all in one fell swoop.

For this reason, we should strive to keep code that deals with a particular data structure contained in as few modules as possible. For instance, if we have a BlogPost database model, it probably makes sense to start out having all the logic regarding a BlogPost in a single file. In that file, we could expose an API allowing consumers to create, publish, edit, delete, update, search, or share blog posts. As the functionality around blog posts grows, we might opt for spreading the logic into multiple colocated files: one might deal with search, parsing raw end-user queries for tags and terms that are then passed to Elasticsearch or some other search engine; another might deal with sharing, exposing an API to share articles via email or through different social media platforms; and so on.

Splitting logic into a few files under the same directory helps us prevent an explosion of functionality that mostly just has a data structure in common, bringing together code that’s closely related in terms of functionality.

The alternative, placing logic related to a particular aspect of our application such as blog posts directly in the components where it’s needed, will cause trouble if left unchecked. Doing so might be beneficial in terms of short-term productivity, but longer-term we need to worry about coupling logic, strictly related to blog posts in this case, together with entirely different concerns. At the same time, if we sprinkle the bulk of the logic across several unrelated components, we risk missing critical aspects of functionality when making large-scale updates to the codebase. We might end up making the wrong assumptions, or mistakes that become evident only much further down the line.

It’s acceptable to start out placing logic directly where it’s needed at first, when it’s unclear whether the functionality will grow or how much. Once this initial exploratory period elapses, and it becomes clear the functionality is here to stay and more might be to come, it’s advisable that we isolate the functionality for the reasons stated previously. Later, as the functionality grows in size and in concerns that need to be addressed, we can componentize each aspect into different modules that are still grouped together logically in the filesystem, making it easy to take all of the interrelated concerns into account when need be.

Now that we have broken down the essentials of module design and how to delineate interfaces, as well as how to lock down, isolate, and drive down complexity in our internal implementations, we’re ready to start discussing JavaScript-specific language features and an assortment of patterns that we can benefit from.