Cross-Functional Teams

Domain-centric teams tend to be cross-functional, meaning every project role is covered by someone on the project. The goal of a domain-centric team is to eliminate operational friction. In other words, the team has all the roles needed to design, implement, and deploy their service, including traditionally separate roles like operations. But these roles must change to accommodate this new structure, which includes the following roles:

Business Analysts

Must coordinate the goals of this service with other services, including other service teams.

Architecture

Design architecture to eliminate inappropriate coupling that complicates incremental change. Notice this doesn’t require an exotic architecture like microservices. A well-designed modular monolithic application may display the same ability to accommodate incremental change (although architects must design the application explicitly to support this level of change).

Testing

Testers must become accustomed to the challenges of integration testing across domains, such as building integration environments, creating and maintaining contracts, and so on.

Operations

Slicing up services and deploying them separately (often alongside existing services and deployed continuously) is a daunting challenge for many organizations with traditional IT structures. Naive old school architects believe that component and operational modularity are the same thing, but this is often not the case in the real world. Automating DevOps tasks like machine provisioning and deployment are critical to success.

Data

Database administrators must deal with new granularity, transaction, and system of record issues.

One goal of cross-functional teams is to eliminate coordination friction. On traditional siloed teams, developers often must wait on a DBA to make changes or wait for someone in operations to provide resources. Making all the roles local eliminates the incidental friction of coordination across silos.

While it would be luxurious to have every role filled by qualified engineers on every project, most companies aren’t that lucky. Key skill areas are always constrained by external forces like market demand. So, many companies aspire to create cross-functional teams but cannot because of resources. In those cases, constrained resources may be shared across projects. For example, rather than have one operations engineer per service, perhaps they rotate across several different teams.

By modeling architecture and teams around the domain, the common unit of change is now handled within the same team, reducing artificial friction. A domain-centric architecture may still use layered architecture for its other benefits, such as separation of concerns. For example, the implementation of a particular microservice might depend on a framework that implements the layered architecture, allowing that team to easily swap out a technical layer. Microservices encapsulate the technical architecture inside the domain, inverting the traditional relationship.

Organized Around Business Capabilities

Organizing teams around domains implicitly means organizing them around business capabilities. Many organizations expect their technical architecture to represent its own complex abstraction, loosely related to business behavior because architect’s traditional emphasis has been around purely technical architecture, that is typically segregated by functionality. For example, a layered architecture is designed to make swapping technical architecture layers easier, not make working on a domain entity like Customer easier. Most of this emphasis was driven by external factors. For example, many architectural styles of the past decade focused heavily on maximizing shared resources because of expense.

Architects have gradually detangled themselves from commercial restrictions via the embrace of open source in all corners of most organizations. Shared resource architecture has inherent problems around inadvertent interference between parts. Now that developers have the option of creating custom-made environments and functionality, it is easier for them to shift emphasis away from technical architectures and focus more on domain-centric ones to better match the common unit of change in most software projects.

Product over Project

One mechanism many companies use to shift their team emphasis is to model their work around products rather than projects. Software projects have a common workflow in most organizations. A problem is identified, a development team is formed, and they work on the problem until “completion,” at which time they turn the software over to operations for care, feeding, and maintenance for the rest of its life. Then the project team moves on to the next problem.

This causes a slew of common problems. First, because the team has moved on to other concerns, bug fixes and other maintenance work is often difficult to manage. Second, because the developers are isolated from the operational aspects of their code, they care less about things like quality. In general, the more layers of indirection between a developer and their running code, the less connection they have to that code. This sometimes leads to an “us versus them” mentality between operational silos, which isn’t surprising, as many organizations have incentivized workers to exist in conflict.

By thinking of software as a product, it shifts the company’s perspective in three ways. First, products live forever, unlike the lifespan of projects. Cross-functional teams (frequently based on the Inverse Conway Maneuver) stay associated with their product. Second, each product has an owner who advocates for its use within the ecosystem and manages things like requirements. Third, because the team is cross-functional, each role needed by the product is represented: business analyst, developers, QA, DBA, operations, and any other required roles.

The real goal of shifting from a project to a product mentality concerns long-term company buy-in. Product teams take ownership responsibility for the long-term quality of their product. Thus, developers take ownership of quality metrics and pay more attention to defects. This perspective also helps provide a long-term vision to the team.

Having small, cross-functional teams also takes andvantage of human nature. Amazon’s “two-pizza team” mimics small group primate behavior. Most sports teams have around 10 players, and anthropologists believe that preverbal hunting parties were also around this size. Building highly responsible teams leverages innate social behavior, making team members more responsible. For example, suppose a developer in a traditional project structure wrote some code two years ago that blew up in the middle of the night, forcing someone in operations to respond to a pager in the night and fix it. The next morning, our careless developer may not even realize they accidentally caused a panic in the middle of the night. On a cross-functional team, if the developer wrote code that blew up in the night and someone from his team had to respond to it, the next morning, our hapless developer has to look across the table at the sad, tired eyes of their team member they inadvertently affected. It should make our errant developer want to be a better teammate.

Creating cross-functional teams prevents finger pointing across silos and engenders a feeling of ownership in the team, encouraging team members to do their best work.

Dealing with External Change

We advocate building components that are highly decoupled in terms of technical architecture, team structure, and so on to allow maximum opportunities for evolution, in the real world, components must interact with one another to share information that collaboratively solves domain problems. So how can we build components that can freely evolve yet make sure we can maintain the integrity of our integration points?

For any dimension in our architecture that requires protection from the side effects of evolution, we create fitness functions. A common practice in microservices architectures is the use of consumer-driven contracts, which are atomic integration architecture fitness functions. Consider the illustration shown in Figure 8-1.

Figure 8-1. Consumer-driven contracts use tests to establish contracts between a provider and consumer(s)

In Figure 8-1, the provider team is supplying information (typically data in a lightweight format) to each of the consumers, C1 and C2. In consumer-driven contracts, the consumers of information put together a suite of tests that encapsulate what they need from the provider and hand off those tests to the provider, who promises to keep those tests passing at all times. Because the tests cover the information needed by the consumer, the provider can evolve in any way that doesn’t break these fitness functions. In the scenario shown in Figure 8-1, the provider runs tests on behalf of all three consumers in addition to their own suite of tests. Using fitness functions like this is informally known as an engineering safety net. Maintaining integration protocol consistency shouldn’t be done manually when it is easy to build fitness functions to handle this chore.

One implicit assumption included in the incremental change aspect of evolutionary architecture is a certain level of engineering maturity amongst the development teams. For example, if a team is using consumer-driven contracts but they also have broken builds for days at time, they can’t be sure their integration points are still valid. Using engineering practice to police practices via fitness functions relieves lots of manual pain from developers but requires a certain level of maturity to be successful.

Connections Between Team Members

Many companies have found anecdotally that large development teams don’t work well, and J. Richard Hackman, a famous expert on team dynamics, offers an explanation as to why. It’s not the number of people but the number of connections they must maintain. He uses the formula shown in Equation 8-1 to determine how many connections exist between people, where n is the number of people.

Equation 8-1. Number of connections between people

In Equation 8-1, as the number of people grows, the number of connections grows rapidly, as shown in Figure 8-2.

Figure 8-2. As the number of people grows, the connections grow rapidly.

In Figure 8-2, when the number of people on a team reaches 20, they must manage 190 links; when it reaches 50 team members, the number of links is a daunting 1225. Thus, the motivation to create small teams revolves around the desire to cut down on communication links. And these small teams should be cross-functional to eliminate artificial friction imposed by coordinating across silos.

Each team shouldn’t have to know what other teams are doing, unless integration points exist between the teams. Even then, fitness functions should be used to ensure integrity of integration points.

Culture

Culture, (n.): The ideas, customs, and social behavior of a particular people or society.

Oxford Dictionary

Architects should care about how engineers build their system and watch out for the behaviors their organization rewards. The activities and decision-making processes architects use to choose tools and create designs can have a big impact on how well software endures evolution. Well-functioning architects take on leadership roles, creating the technical culture and designing approaches for how developers build systems. They teach and encourage in individual engineers the skills necessary to build evolutionary architecture.

An architect can seek to understand a team’s engineering culture by asking questions like:

• Does everyone on the team know what fitness functions are and consider the impact of new tool or product choices on the ability to evolve new fitness functions?

• Are teams measuring how well their system meets their defined fitness functions?

• Do engineers understand cohesion and coupling?

• Are there conversations about what domain and technical concepts belong together?

• Do teams choose solutions not based on what technology they want to learn, but based on its ability to make changes?

• How are teams responding to business changes? Do they struggle to incorporate small changes, or are they spending too much time on small business change?

Adjusting the behavior of the team often involves adjusting the process around the team, as people respond to what is asked of them to do.

Tell me how you measure me, and I will tell you how I will behave.

Dr. Eliyahu M. Goldratt (The Haystack Syndrome)

If a team is unaccustomed to change, an architect can introduce practices that start making that a priority. For example, when a team considers a new library or framework, the architect can ask the team to explicitly evaluate, through a short experiment, how much extra coupling the new library or framework will add. Will engineers be able to easily write and test code outside of the given library or framework, or will the new library and framework require additional runtime setup that may slow down the development loop?

In addition to the selection of new libraries or frameworks, code reviews are a natural place to consider how well newly changed code supports future changes. If there is another place in the system that will suddenly use another external integration point, and that integration point will change, how many places would need to be updated? Of course, developers must watch out for overengineering, prematurely adding additional complexity or abstractions for change. The Refactoring book contains relevant advice:

Many teams are driven and rewarded most often for delivering new functionality, with code quality and the evolvable aspect considered only if teams make it a priority. An architect that cares about evolutionary architecture needs to watch out for team actions that prioritize design decisions that help with evolvability or to finds ways to encourage it.

Culture of Experimentation

Successful evolution demands experimentation, but some companies fail to experiment because they are too busy delivering to plans. Successful experimentation is about running small activities on a regular basis to try out new ideas (both from a technical and product perspective) and to integrate successful experiments into existing systems.

The real measure of success is the number of experiments that can be crowded into 24 hours.

Thomas Alva Edison

Organizations can encourage experimentation in a variety of ways:

Bringing ideas from outside

Many companies send their employees to conferences and encourage them to find new technologies, tools, and approaches that might solve a problem better. Other companies bring in external advice or consultants as sources of new ideas.

Encouraging explicit improvement

Toyota is most famous for their culture of kaizen, or continuous improvement. Everyone is expected to continually seek constant improvements, particularly those closest to the problems and empowered to solve them.

Spike and stabilize

A spike solution is an extreme programming practice where teams generate a throw-away solution to quickly learn a tough technical problem, explore an unfamiliar domain, or increase confidence in estimates. Using spike solutions increases learning speed at the cost of software quality; no one would want to put a spike solution straight into production because it would lack the necessary thought and time to make it operational. It was created for learning, not as the well engineered solution.

Creating innovation time

Google is well known for their 20% time, where employees can work on any project for 20% of their time. Other companies organize Hackathons and allow teams to find new products or improvements to existing products. Atlassian holds regular 24-hour sessions called ShipIt days.

Following set-based development

Set-based development focuses on exploring multiple approaches. At first glance, multiple options appear costly because of extra work, but in exploring several options simultaneously, teams end up with a better understanding of the problem at hand and discover real constraints with tooling or approach. The key to effective set-based development is to prototype several approaches in a short time-period (i.e., less than a few days) to build more concrete data and experience. A more robust solution often appears after taking into account several competing solutions.

Connecting engineers with end-users

Experimentation is only successful when teams understand the impact of their work. In many firms with an experimentation mindset, teams and product people see first-hand the impact of decisions on end-customers and are encouraged to experiment to explore this impact. A/B testing is one such practice companies use with this experimentation mindset. Another practice companies implement is sending teams and engineers to observe how users interact with their software to achieve a certain task. This practice, taken from the pages of the usability community, builds empathy with end-users and engineers often return with a better understanding of user needs, and with new ideas to better fulfill them.

Case Study: PenultimateWidgets as a Platform

Business at PenultimateWidgets is going so well they have decided to sell part of their platform to other sellers of things like widgets. Part of the appeal of the PenultimateWidgets platform is its proven scalability, resiliency, performance, and other assets. However, their architects don’t want to sell the platform only to start hearing stories of failures because users extend it in damaging ways.

To help preserve the important characteristics of the platform, the PenultimateWidgets architects provide a deployment pipeline along with the platform with built-in fitness functions around important dimensions. To remain certified, users of the platform must preserve the existing fitness functions and (hopefully) add their own as they extend the platform.

Low-Hanging Fruit

If an organization needs an early win to prove the approach, architects may choose the easiest problem that highlights the evolutionary architecture approach. Generally, this will be part of the system that is already decoupled to a large degree and hopefully not on the critical path to any dependencies. Increasing modularity and decreasing coupling allows teams to demonstrate other aspects of evolutionary architecture, namely fitness functions and incremental change. Building better isolation allows more focused testing and the creation of fitness functions. Better isolation of deployable units makes building deployment pipelines easier and provides a platform for building more robust testing.

Metrics are a common adjunct to the deployment pipeline in incremental change environments. If teams use this effort as a proof-of-concept, developers should gather appropriate metrics for both before and after scenarios. Gathering concrete data is the best way to for developers to vet the approach; remember the adage that demonstration defeats discussion.

This “easiest first” approach minimizes risk at the possible expense of value, unless a team is lucky enough to have easy and high value align. This is a good strategy for companies that are skeptical and want to dip their toes in the metaphorical water of evolutionary architecture.

Highest-Value

An alternative approach to “easiest first” is “highest value first”—find the most critical part of the system and build evolutionary behavior around it first. Companies may take this approach for several reasons. First, if architects are convinced that they want to pursue an evolutionary architecture, choosing the highest value portion first indicates commitment. Second, for companies still evaluating these ideas, their architects may be curious as to how applicable these techniques are within their ecosystem. Thus, by choosing the highest value part first, they demonstrate the long-term value proposition of evolutionary architecture. Third, if architects have doubts that these ideas can work for their application, vetting the concepts via the most valuable part of the system provides actionable data as to whether they want to proceed.

Testing

Many companies lament the lack of testing their systems have. If developers find themselves in a code base with anemic or no testing, they may decide to add some critical tests before undertaking the more ambitious move to evolutionary architecture.

It is generally frowned upon for developers to undertake a project that only adds tests to a code base. Management looks upon this activity with suspicion, especially if new feature implementation is delayed. Rather, architects should combine increasing modularity with high-level functional tests. Wrapping functionality with unit tests provides better scaffolding for engineering practices such as test-driven development (TDD) but takes time to retrofit into a code base. Instead, developers should add coarse-grained functional tests around some behavior before restructuring the code, allowing you to verify that the overall system behavior hasn’t changed because of the restructuring.

Testing is a critical component to the incremental change aspect of evolutionary architecture, and fitness functions leverage tests aggressively. Thus, at least some level of testing enables these techniques, and a strong correlation exists between comprehensiveness of testing and ease of implementing an evolutionary architecture.

Infrastructure

New capabilities come slow to some companies, and the operations group is a common victim of lack of innovation. For companies that have a dysfunctional infrastructure, getting those problems solved may be a precursor to building an evolutionary architecture. Infrastructure issues come in many forms. For example, some companies outsource all their operational responsibilities to another company and thus don’t control that critical piece of their ecosystem; the difficultly of DevOps rises orders of magnitude when saddled with the overhead of cross-company coordination.

Another common infrastructure dysfunction is an impenetrable firewall between development and operations, where developers have no insight into how code eventually runs. This structure is common in companies rife with politics across divisions, where each silo acts autonomously.

Lastly, architects and developers in some organizations have ignored good practices and consequently built massive amounts of technical debt that manifests within infrastructure. Some companies don’t even have a good idea of what runs where and other basic knowledge of the interactions between architecture and infrastructure.

Ultimately, the advice parallels the annoying-but-accurate consultant’s answer of It Depends! Only architects, developers, DBAs, DevOps, testing, security, and the other host of contributors can ultimately determine the best roadmap toward evolutionary architecture.

Case Study: Enterprise Architecture at PenultimateWidgets

PenultimateWidgets is considering revamping a major part of their legacy platform, and a team of enterprise architects generated a spreadsheet listing all the properties the new platform should exhibit: security, performance metrics, scalability, deployability, and a host of other properties. Each category contained 5 to 20 cells, each with some specific criteria. For example, one of the uptime metrics insisted that each service offer five nines (99.999) of availability. In total, they identified 62 discrete items.

But they realized some problems with this approach. First, would they verify each of these 62 properties on projects? They could create a policy, but who would verify that policy on an ongoing basis? Verifying all these things manually, even on an ad hoc basis, would be a considerable challenge.

Second, would it make sense to impose strict availability guidelines across every part of the system? Is it critical that the administrator’s management screens offer five nines? Creating blanket policies often leads to egregious overengineering.

To solve these problems, the enterprise architects defined their criteria as fitness functions and created a deployment pipeline template each project starts with. Within the deployment pipeline, the architects designed fitness functions to automatically check critical features such as security, leaving individual teams to add specific fitness functions (like availability) for their service.

Fitness Functions Using AI

Gradually, large open source artificial intelligence frameworks are becoming available for regular projects. As developers learn to utilize these tools to support software development, we envision fitness functions based on AI that look for anomalous behavior. Credit card companies already apply heuristics such as flagging near-simultaneous transactions in different parts of the world; architects can start to build investigatory tools to look for odd behaviors in architecture.

Generative Testing

A practice common in many functional programming communities gaining wider acceptance is the idea of generative testing. Traditional unit tests include assertions of correct outcomes within each test case. However, with generative testing, developers run a large number of tests and capture the outcomes then use statistical analysis on the results to look for anomalies. For example, consider the mundane case of boundary checking ranges of numbers. Traditional unit tests check the known places where numbers break (negatives, rolling over numerical sizes, and so on) but are immune to unanticipated edge cases. Generative tests check every possible value and report on edge cases that break.

Why Should a Company Decide to Build an Evolutionary Architecture?

Many businesses find that the cycle of change has accelerated over the past few years, as reflected in the aforementioned Forbes observation that every company must be competent at software development and delivery.

Case Study: Selective Scale at PenultimateWidgets

PenultimateWidgets has some services that require little in the way of scale and are therefore written in simple technology stacks. However, a couple of services stand out. The Import service must import inventory figures from brick-and-mortar stores every night for the accounting system. Thus, the architectural characteristics and fitness functions the developers built into Import include scalability and resiliency, which greatly complicate the technical architecture of that service. Another service, MarketingFeed, is typically called by each store at opening to get daily sales and marketing updates. Operationally, MarketingFeed needs elasticity to be able to handle the burst of requests as stores open across time zones.

A common problem in highly coupled architectures is inadvertent overengineering. In a more coupled architecture, developers would have to build scalability, resiliency, and elasticity into every service, complicating the ones that don’t need those capabilities. Architects are accustomed to choosing architectures against a spectrum of tradeoffs. Building architectures with clearly defined quantum boundaries allows exact specification of the required architectural characteristics.

Why Would a Company Choose Not to Build an Evolutionary Architecture?

We don’t believe that evolutionary architecture is the cure for all ailments! Companies have several legitimate reasons to pass on these ideas.

Convincing Others

Architects and developers struggle to make nontechnical managers and coworkers understand the benefits of something like evolutionary architecture. This is especially true of parts of the organization most disrupted by some of the necessary changes. For example, developers who lecture the operations group about doing their job incorrectly will generally find resistance.

We introduced the best solution to this problem in Chapter 6. Rather than try to convince reticent parts of the organization, demonstrate how these ideas improve their practices.

Case Study: Consulting Judo

A colleague was working with a big retailer trying to convince the enterprise architects and operations group to embrace more modern DevOps practices, such as automated machine provisioning, better monitoring, and so on. Yet her pleas fell on deaf ears because of two common refrains: “We don’t have time” and “Our setup is so complex, those things will never work here.”

She applied an excellent technique called consulting judo. Judo as a martial art has numerous techniques that use the opponent’s weight against them. Consulting judo entails finding a particular pain point and fixing it as an exemplar. The pain point at the retailer was QA environments: There were never enough of them. Consequently, teams would attempt to share environments, but that caused major headaches. Having found her case study, she received approval to creating QA environments using modern DevOps tools and techniques.

When she was complete, she demonstrated the falseness of both previous assumptions. Now, any team that needs a QA environment can provision one trivially. Her effort in turn convinced operations to invest more fully into modern techniques because of demonstrable value. Demonstration defeats discussion.

“The Future Is Already Here…”

The future is already here—it’s just not very evenly distributed.

William Gibson

Many companies view software as overhead, like the heating system. When software architects talk to executives at those companies about innovation in software, they imagine plumbers upselling them on pretty but expensive overhead. However, that antiquated view of the strategic importance of software is discredited. Consequently, decision makers who control software purchases tend to became institutionally conservative, valuing cost savings over innovation. Enterprise architects make this mistake for understandable reasons—they look at other companies within their ecosystem to see how they approach these decisions. But that approach is dangerous because a disruptive company that has modern software architecture may move into the existing company’s realm and suddenly dominate because they have better information technology.

Moving Fast Without Breaking Things

Most large enterprises complain about the pace of change within the organization. One side effect of building an evolutionary architecture manifests as better engineering efficiency. All the practices we call incremental change improve automation and efficiency. Defining top-level enterprise architecture concerns as fitness functions both unifies a disparate set of concerns under one umbrella and forces developers to think in terms of objective outcomes.

Building an evolutionary architecture implies that teams can make incremental changes at the architectural level with confidence. In Chapter 2, we described a GitHub case study where a foundational component of an architecture with no regressions (while uncovering other undiscovered bugs). Business people fear breaking change. If developers build an architecture that allows incremental change with better confidence than older architectures, both business and engineering win.

Less Risk

With improved engineering practices comes decreased risk. Evolutionary architecture forces modern practices on teams in the guise of incremental change, a beneficial side effect. Once developers have confidence that their practices will allow them to make changes in the architecture without breaking things, companies can increase their release cadence.

New Capabilities

The best way to sell the ideas of evolutionary architecture to the business revolves around the new business capabilities it delivers, such as hypothesis-driven development. Business people glaze over when architects wax poetic about technical improvements, so it is better to couch the impact in their terms.