Considering Goals and Objectives

Application testing can’t succeed unless the development team defines both goals and objectives for the testing. A goal is as simple as determining whether the application meets the technical and business requirements that an organization requires of it. An objective is to determine that the application can successfully perform a set of tasks within the business environment provided for it. For example, an objective might be to add new users to a database without causing errors, duplicating users, or leaving out essential information. The following sections discuss goals and objectives that testing must meet in order to provide a useful result.

Testing performance

Many people equate performance with speed, but performance encompasses more than simply speed. An application that is fast, but performs the task incorrectly, is useless. Likewise, an application that performs a task well, but makes the information it processes available to the wrong parties, is also useless. In order to perform well, an application must perform tasks reliably, securely, and quickly.

Each of these performance elements weighs against the others. Increasing security will necessarily make the application slower because the developer adds more code to perform security checks. Adding code makes the application run more slowly. Likewise, reliability will cause an application to slow because more checks are added in this case as well. Security checks can decrease application reliability by reducing risk at the expense of functionality—a reliable application is one that provides all the expected functionality in every given circumstance (it doesn’t fail). In short, all elements of performance work against each other, as shown in Figure 11-1.

Figure 11-1. Performance encompasses speed, reliability, and security

In order to test the performance of an application, you must verify the balance between speed, reliability, and security. Balanced applications perform well and don’t place a burden on the user, but still handle data reliably and efficiently.

Testing Internal Libraries

When testing internal libraries—those that you control and own—you have access to the source code and can perform unit testing on each of the elements before you even put the library together. This approach gives you additional control over the testing process, lets you test and fix bugs when the costs are lower, and ensures you can perform both static and dynamic testing in order to locate potential problems with greater efficiency.

Of course, the fact that you own the code also means that you’re writing the library at the same time as the rest of the application. Other developers will want to perform testing as well. What this means is that you must employ mocking, as described in Chapter 10, to ensure that development can progress as anticipated. As you perform individual unit testing and are certain that the library elements work as anticipated, you can replace the mocked elements with real elements.

As part of the testing process, you can also create a testing harness for the library as described in “Creating a Test Harness for Libraries”. However, instead of creating the whole test harness at once, you create it a piece at a time as the real library elements become available. Creating the test harness in this manner will help you keep track of how library elements are interacting with the application and make changes as needed (when the cost of doing so is low).

Testing Internal APIs

As with internal libraries, you own the code for an internal API. Because an API isn’t part of the application—it runs in a different process—you need to create a server setup for interacting with the API. However, you won’t want to use a production server for the task because the code you create will have all sorts of issues. It pays to configure the application such that you can change just one configuration item to point the application to the production API once you release the API.

Any API you create will also need to rely on mocking so that application developers can begin writing the application code while you continue to work on the API. As the real code becomes available, you need to replace the mocked elements with real elements. It pays to develop the whole set of test scripts for an API at the outset using the techniques found in “Creating Testing Scripts for APIs”, realizing that the mocked elements will provide canned responses. In fact, the canned responses will clue you in as to which elements are still mocked.

It’s essential to test your internal APIs using the same criteria as the external APIs and to configure the testing and development environments to match your production environment. Otherwise, you could end up with a situation where an internal API actually becomes a pathway for a hacker to gain access to your network. Even disconnected software is subject to various kinds of hacks (as described in previous chapters).

Testing External Libraries

An external library (including frameworks and all sorts of other library-like coding structures) is one that someone else owns. The library is complete before you even begin writing your application and theoretically, the third party tests and maintains the library code. However, the library is simply a black box in most respects. Your ability to perform a full static test is limited to the public-facing modules that the third party makes available. Due to the complexities of working with most third-party libraries, a full static test is out of the question, which leaves dynamic testing.

Before you begin writing your application, during the proposal and design stages, you need to ensure any third-party library you choose is safe and fully meets the criteria for your application. The most popular libraries provide you with a test harness or you can find a third-party test harness for them. However, when working with a smaller, less known library, you need to set up testing yourself.

Testing External APIs

External APIs are popular precisely because someone else owns the code and it doesn’t even run on the local system. You won’t download the code or do anything with it except to make calls. The siren call of external APIs lulls even the careful developer into a false sense of security. Previous chapters have told you all about the terrifying potential for APIs to cause you woe. If anything, you need to script tests for external APIs with even greater care than any other code you use because unlike external libraries (and by extension, frameworks) you won’t ever see the code. There is no possibility whatsoever of performing a static test so your dynamic tests had better run the API through its courses before you make a decision about using it.

Unlike libraries, it’s unlikely that you’ll find a ready-made scripting suite for an API. In order to verify that the API works as anticipated, you really do need to create a scripting setup and send inputs of all sorts to the API. It’s essential to track the responses you receive, especially to errant inputs. You don’t know how an API will respond to errant inputs. Consequently, you don’t know how to code your application to react to errant input feedback. In other words, you need to know how the API will react when it receives data that is out of range or potentially of the wrong type.

The assumption of most developers is that the errant input will come from application users. However, errant input to the API can come from a botched man-in-the-middle attack or from other sorts of hacks. The errant input could also come from other sources on your system that reflects some type of infection or other problem. By being aware of how an API reacts to errant input, you can create a sort of security indicator that tells you something is wrong. Consider it a canary in the mine strategy. Errant inputs don’t just happen in most cases—there is a cause and knowing the kind of errant input that provides an unexpected response can provide clues as to the source of an issue.

Extending Testing to Microservices

You test microservices using the same techniques as you do APIs. As with APIs, you only have the ability to perform dynamic testing unless you happen to own the microservice code. In addition, it’s essential to track responses to unexpected data inputs, especially when you plan to use multiple microservices to perform the same task (with the alternatives providing backup to a main microservice that you select). The responses you receive may vary between microservices, which will mean your error handling code becomes trickier.

The biggest issue to consider with microservices, however, is that the developer purposely keeps microservices small. You can’t test the most commonly used features because every feature is commonly used. In short, your testing scripts must now test every microservice fully, which could add a burden to the testing group.

Creating a Test Harness for Libraries

A test harness is a set of instructions within the application code or as part of a special addition to the code that performs various tests. Because libraries exist as part of the application, the instructions for testing the library also appear as part of the application.

Test instructions normally appear in debug code as some sort of assert() function, or by making use of logging or screen outputs. JavaScript lacks an assert() function (there is talk of adding one). However, you can use the error object to create an assert-like function that provides the same sort of information. In using an assert() setup, you create assertions in the code that look something like this:

assert(typeof myArg === "string");

The assert() function would look something like this:

function assert(condition, message)
{
    if (!condition)
    {
        message = message || "Assertion failed";
        if (typeof Error !== "undefined")
        {
            throw new Error(message);
        }
        else
        {
            throw message;
        }
    }
}

In this case, when the value of an argument or some other code condition fails, the test ends with an error. You can choose to log the error or work with it in other ways, but you know the test failed. Testing the condition isn’t a problem with any browser. However, you may find different browsers support the error object in different ways, so simply throwing the message (rather than re-throwing the error object) is a good fallback position.

The in-code element does perform tests, but you still need input for those tests. To obtain this part of the puzzle, you normally need to employ scripted manual testing, which is always error prone, or a third-party product to script the required inputs. The tests you run against the library will tell you precisely how well the library meets the requirements of your application.

Creating Testing Scripts for APIs

API support consists of making calls. It’s possible to test an API without even using the application. All you need is a script that makes calls and checks responses. A number of third-party products perform this task or you can create a simple application to perform the testing manually. Using a script ensures you get the same testing results each time, so using a scripting product is usually the best choice. Any test you create should check for these conditions as a minimum:

Range

Responds correctly within its range and provides a correct error response when values are either too high or low.

Type

Verifies that the user supplies the right data type as input and provides a correct error response when the input is of the wrong type.

Size

Validates the length of the data so that it’s not possible for someone to send a script instead of the expected string.

Characters

Tests for invalid characters within the input to ensure the user can’t send control or other incorrect characters as part of the input.

After you start using the API from the application, you need to perform integration testing, which consists of providing inputs to the application and then requiring the application to make the required calls. Again, you can use a scripting product to make this task easier.

Extending Testing Strategies to Microservices

As with APIs, you want to use some type of scripting product to make calls and check responses when working with a microservice. However, you need to ensure that the microservice is checked thoroughly because each one represents a separate piece of code. It’s not possible to make assumptions about a microservice in the same way that you can with an API.

When you do perform integration testing, you need to determine the performance profile of the application. Every microservice should receive at least one test. However, microservices you plan to use more often should receive more testing. The purpose of this strategy is to verify code that has a higher probability of causing problems and to keep testing costs lower.

Developing Response Strategies

The focus of all testing is on the response provided by the library, API, or microservice to a given input. Unless the code reacts in the proper manner, the application won’t work as originally envisioned. More importantly, hackers look for discrepancies in behavior to exploit. When defining responses, you must consider two response types: direct and mocked. The following sections discuss each response type.

Devising Tests for HTML Issues

When working with HTML, you need to consider that the language provides the basic user interface and that it also provides the avenue where many user-specific security issues will take place. With this in mind, you need to ensure that the HTML used to present information your application manages is tested to ensure it will work with a wide variety of browsers. With this in mind, here are some language-specific issues to consider:

• The HTML is well formed and doesn’t rely on tags or attributes that aren’t supported by most browsers.

• The document is encoded correctly.

• Any code within the document performs required error handling and checks for correct input.

• The document output looks as expected when provided with specific inputs.

• Selection of user interface elements reduces the potential for confusion and errant input.

Devising Tests for CSS Issues

The original intent for CSS was to create a means for formatting content that didn’t involve the use of tables and other tricks. The problem with these tricks is that no one used a standard approach, and the tricks tended to make the page unusable for people with special needs. However, CSS has moved on from simple formatting. People have found ways to create special effects with CSS. In addition, CSS now almost provides a certain level of coding functionality. As a result, it has become important to test CSS just as fully as you do any other part of the application. CSS has the potential for hiding security issues from view. With this in mind, you need to perform CSS-specific tests as part of the security testing for your application—it should meet the following criteria:

• The CSS is well formed.

• There aren’t any nonstandard elements in the code.

• Special effects don’t cause accessibility problems.

• The choice of colors, fonts, and other visual elements reflect best practice for people with special needs.

• It’s possible to use an alternative CSS format when the user has special needs to address.

• Given an event or particular user input, the CSS provides a consistent and repeatable output effect.

Devising Tests for JavaScript Issues

JavaScript will provide most of the functional code for your application. With this in mind, you test JavaScript code using the same approach as you would other programming languages. You need to verify that for a given input, you get a specific output. Here are some issues you need to consider as part of your testing suite:

• Ensure the code follows the standards.

• Test the code using a full range of input types to ensure it can handle errant input without crashing.

• Perform asynchronous testing to ensure that your application can handle responses that arrive after a nondeterministic interval.

• Create test groups so that you can validate a number of assertions using the sample assertion code found in “Creating a Test Harness for Libraries” (or an assert() provided as part of a test library). Test groups magnify the effect of using assertions for testing.

• Verify that the code is responsive.

• Check application behavior to ensure that a sequence of steps produces a desired result.

• Simulate failure conditions (such as the loss of a resource) to ensure the application degrades gracefully.

• Perform any testing required to ensure the code isn’t susceptible to recent hacks that may not be fixed on the client system. Using a security-specific analyzer, such as VeraCode, can help you locate and fix bugs that might provide entry to hackers based on recently found security issues.

One of the issues you need to research as part of your specification and design stages is the availability of existing test suites. For example, the ECMAScript Language test262 site can provide you with some great insights into your application.