Defining the Reasons for Hiring the Third Party

There are many reasons you might have for hiring a third party to perform application testing for you. The reasons vary by organization, application type, and the problem domain the application solves. Before you hire a third party to perform testing, it’s important to know why you’re hiring the third party. If you don’t have this piece of essential information, the testing effort is doomed from the start because you have unstated and potentially unrealistic expectations of the testing process. In short, you can’t know that the testing is a success because you have no idea of what you expected in the first place. With this in mind, the most common reasons for hiring a third party are:

Quality

Using someone who specializes in testing can improve the quality of your software because an organization devoted to testing will know all the latest techniques for performing tests in your particular problem domain. These testers also perform testing tasks every day, so they know about potential bugs and issues that commonly occur with software that uses a particular programming language, operating system, platform, and addresses specific user needs. It’s also quite possible that the quality of the reports you receive from the testing process will be better than anything produced by your own organization because testers fill out this type of report every day.

Cost

A professional tester can perform the testing tasks faster and with greater accuracy than someone who performs the task on a part-time basis. The tester can also avoid tests that won’t provide anything with your particular application—a problem that occurs regularly within organizations (wasting both time and money). In addition, it’s possible to outsource the testing to a location in the world where the actual costs of hiring someone to perform this task is much lower than in your location.

Training

A hidden cost that many organizations don’t consider is the time and expense of training a team to perform the testing tasks. Even if team members perform the task part time, they won’t have the experience that professional testers do and may need additional training for each new kind of application tested. The only time that it pays to have an actual testing team is if your organization writes enough software to keep the team engaged in testing full time.

Time

Setting up a testing lab, training the testers, and waiting for the testers to become proficient in performing the required work takes time that many organizations simply don’t have. In order to get a product to market on time, it becomes necessary to test during the writing process, using techniques that are as efficient as possible, on systems that match the production system as closely as possible, and with reports back to developers that help locate and fix problems as quickly as possible.

Expertise

Your organization may have recently made an update in platform, equipment, or other resources that require another cycle of testing. The application or related services haven’t changed, but the environment in which they operate has. If you find that the newness of the setup causes problems for your existing testing team, it may be time to get third-party help in discovering what these changes mean to the software and to get new checks established. The third party can also provide much needed training.

Considering the Range of Possible Testing Services

The essential benefit of using a third-party tester is ensuring that you get full value out of the services provided. However, the best testing service in the world can’t possibly know everything about every possible sort of testing. You need to define precisely what sort of testing you want done and then use these specifics when choosing a testing company. As a first step, you need to define the environment:

• Platform

• Operating system

• Programming language

• Problem domain

• Organization type

• User type

Not every testing company performs every sort of testing available. Once you get past the environmental factors, you need to consider the sorts of testing you need done. To get the best results, it’s important that the third-party tester provide facilities for performing all the tests you need. Here are the most common types:

• Security

• Functional

• Automation

• Performance

• Integration

Some testing companies offer just one thing—testing. However, you may decide that you need other services in order to succeed in your application development task. A full service company typically charges a lot more than one that offers just testing. The number of services doesn’t always affect the bottom line, but it can and you need to choose just those services you actually need. Here are some common add-on services:

Developer coaching

In some cases, the testing problem may turn up issues that your developers simply haven’t encountered. The use of developer coaching helps your team locate and fix problems faster. In addition, it provides a level of training that is specific to your application, which means that the information learned is organization-specific and focused (increasing its value).

Program management

You may need some help managing the testing effort. Just as your development team may lack testing experience, the management team may lack experience in managing certain aspects of the product lifecycle. Some testing companies can help your organization overcome this problem.

eLearning

An organization as a whole may require some additional training to make the application work better. For example, many users are unaware of the issues surrounding security, so they make mistakes that could cause a security breach. The use of eLearning modules helps people discover the skills needed to use the application successfully during times when the user actually has the motivation and availability to complete the training.

It’s also important to consider the specifics of the security testing for your application. The third-party tester may offer some types of security testing, but not others. In addition, you need to know what the third party should provide as part of the service. Here are some things to consider:

Environment

You need to define whether the testing is against staging or production environments. Staging environment testing works best during the early development stage of the application. Production environment testing works best during the final development stages and when the application is first released to the production environment.

Testing level

Some testers perform a basic suite of tests to ensure that the security risks for your application remain low. However, this basic testing might not tell you enough to fix the problems your application contains. You may want the test to provide full proof of concept exploits, with sample code, so you can see precisely how the hacks work in order to provide a better fix. (The third party can sometimes help perform triage on your application as well.)

Resource usage

Testing uses resources on your system. In order to prevent loss of application functionality or availability during testing, you need to specify resource usage rules. The third-party tester can configure testing tools to remain within whatever guidelines you set. In addition, make sure you specify resource usage as a function of day and time. For example, you may want the third-party tester to focus testing on evenings or weekends when the system under test receives less use and you can relax some of the resource usage rules.

Testing stop conditions

You don’t want security testing to result in a data breach or other security issue. In some cases, security testing could actually result in a security issue. Defining stop conditions helps prevent the testing scenario from turning into a situation where you actually need to perform data recovery or make a public statement about a data breach.

Ensuring the Third Party Is Legitimate

Anyone can come to you saying that their company provides superior testing services in a timely manner and at the lowest possible cost. It doesn’t mean that they actually deliver on their promises. What it could mean is that they have a great marketing staff that can write above-average material, but that none of the write-up will ever come to fruition. A bad third-party tester could cost you everything by allowing a competitor to deliver product first, causing a data breach that drives customers away, or makes it impossible for people in your organization to perform their jobs. Consequently, ensuring the vendor can actually deliver on promises made is an essential part of finding a vendor.

Security testing is one of those areas where you really don’t want to take any chances, so ensure the third party you hire has all of the required certifications. The American Software Testing Qualifications Board (ASTQB) is a good place to start looking for the certifications the third party requires. Another place to look is the International Software Certifications Board.  No matter which group you choose to support, make sure the company you hire has testers with the appropriate certifications. In addition, get guarantees in writing that only the certified testers will work on your project (or else the company might substitute substandard help that is marginally supervised by those who possess the certifications).

It also pays to look at vendor stability. An annual report can tell you a lot about the health of an organization, which in turn tells you whether the organization is successful in completing tasks. Successful organizations tend to do a better job of testing and you want the best for your application. You should also dig into organization statistics such as the amount of turnover (which indicates employee happiness) and determine whether the organization has ever had bad press attached to its name.

Reviews from previous customers can help. However, it’s essential to remember that reviews are simply opinions and customers are extremely biased at times. In addition, an organization can seed positive reviews to make it appear that it provides better service than it really does. Check out the reviews and ensure that the customer actually had work done by the company when possible. Contact the customer to see what sort of testing it had done (again, if possible).

Interviewing the Third Party

At this point, you know precisely why you want third-party help, that you have found a company that provides the services you need, and that the company is legitimate. It’s time to have a discussion with the people in the company and determine whether you can create a working relationship with them. This isn’t like an interview for hiring someone for your organization—you’re hiring an entire company to perform testing on your application in order to ensure that the application is secure. This means getting all the stakeholders in your company involved with the interview and determining the best candidate for the job. The stakeholders include management, of course, but they also include developers, trainers, users, and anyone else who is affected by the application. The question you need to answer is whether the company you’re hiring really can do the required levels of testing to obtain a usable application. If you need other services, then you also need to interview the company about the support you can expect (which is where the users could excel if the company is providing training support).

Performing Tests on a Test Setup

Before you sign on the dotted line, make sure you know that the company you’re hiring can do the work. Have the company provide some sort of limited demonstration that walks through a generalized testing process if at all possible. You should be able to see a walkthrough or some other evidence that the company can actually do the work. Otherwise, it’s really hard to know what you’re getting until it’s too late to back out of the deal. Some demonstration of proficiency is always a good idea, even if it’s a dry run-through of the process the company plans to employ.

Specifying the Third-Party Goals in Testing

Chapter 9 discusses the need for goals in application testing. Chapter 11 further emphasizes the need for goals and describes them in some detail. However, these are goals for your application as a whole and your organization as a whole. A third-party tester is a partner in the testing process, which means that it only works through a part of the process. The goals of a third-party tester are similar to your goals in that the testing must still reflect specific principles; however, the goals now also include the need to communicate all testing criteria and methodologies clearly. In addition, you must provide goals that include the concepts of your organization’s management strategy. Consequently, the goals described in Chapter 9 and Chapter 11 are a good starting point, but you must now augment them to ensure the third party knows precisely what is required and how to respond to requests for reporting information.

Generating a Written Test Plan

You always need to create a written test plan. However, when working with a third-party tester, you must go over the written test plan to ensure the third-party testing roles are clear and that the third party completely understands all the testing requirements. Where questions exist, the third-party tester should be able to provide input that helps clarify the testing requirements. In fact, this input is invaluable as training for future test plans. A third-party tester should be able to help you create a test plan that fully adheres to industry best practices and helps you obtain the precise results you want from the testing process.

As part of the testing plan, you need to list the tests to conduct using the third-party tester’s list of services as input. In other words, the test plan should clearly state which of the third-party tester services are part of the testing process so that no ambiguity exists once testing starts. Make sure you understand precisely which tests are run and why these tests are necessary to ensure the application works as anticipated. A third-party tester must articulate the need for the tests and other services clearly. Otherwise, the testing process could fail due to a lack of communication.

Enumerating the Test Output and Reporting Requirements

The test plan must contain a section that describes the expected outputs precisely. In addition, the test plan must detail what to do when tests fail so that the response is organized and well considered before the testing process begins. The third-party tester can usually help provide additional information about expected outputs and potentially additional individual test types so that you can verify that your application truly is secure against the latest threats. It’s unlikely that your own staff will know about every potential threat, so part of the third-party tester’s job is to ensure you test for security issues that you don’t know about in addition to those you do know about. The test plan should include detailed outputs for every test.

When a test cycle is complete or some other event occurs, the test plan should specify some type of written report. The report need not appear on paper, but simply in a permanent form that everyone can review and study as needed. The reports are usually tied to tests or other significant events and the test plan should detail precisely which tests generate specific reports.

Considering Test Requirements

Tests are more than simple inputs and outputs conducted in a specific environment. As part of your test plan, you must consider how the testing company performs the tests. It can become a sticking point in some situations. For example, you may have only the United States in mind for a market. If your testing company is in Hong Kong, they’re used to speaking UK English. The differences are small, but enough to make testing less useful in some situations. Consider parsing words that vary slightly between the two English forms, such as color and colour. Your code may not parse colour correctly. The reverse also holds true. A testing company needs to support all the languages you intend to support.

Considering how you test an application is important from all sorts of angles. For example, if your application is designed to meet specific healthcare requirements and the testing company doesn’t place enough emphasis on accessibility testing, the application may provide everything but the essential of making access easy. It doesn’t matter if the application is secure when the user can’t interact with it in the first place.

Creating this portion of the testing plan is especially difficult because you have already trained any testing staff in your organization to look for the special requirements of your application. It’s easy to make assumptions about how the testing company will view testing based on what you already do. The key issue is to assume nothing when it comes to testing—write everything down.

Determining Organizational Participation in Testing

It’s important to involve your organization in the testing process. Otherwise, you can’t be sure the testing company is performing the task correctly. However, monitoring the testing process differs substantially from hands-on participation. You may even choose to divide the testing process into functional areas—those that your organization already has the expertise to perform and those that the testing company specializes in. The testing plan should spell out precisely what level of participation each entity provides.

Implementing the test plan often shows that the best plans on paper still lack real-world usefulness. You may find that the test plan doesn’t include every area of participation (leaving areas to split after the fact) or that it fails to consider the logistics of performing the tests in the manner specified. The test plan may not define the required level of communication adequately or you might find that the communication process is chatty, which wastes time. Performing a dry run of the testing process usually tells you about these issues before they become a problem.

Beginning the Testing Process

After all the planning and dry runs, it’s time to start the testing process. You may try to psyche yourself into believing the first day will go well, but you’ll be disappointed. It’s smarter to assume that some problems will occur during the testing process and that you need to deal with them up front. What this means to the developer is communicating issues such as missing source code files to the testing organization as quickly as possible. You may also find that the testing company can’t access online resources, such as libraries, APIs, and microservices, required to run the application. The best idea is to set the first few days of testing aside to work out kinks in the system so that a general rush to get going doesn’t compound the issues you confront during startup.

Depending on your application configuration, purpose, and resources, you should also plan to address a number of security issues. The testing company may not have access to paid online resources or you might have to provide additional rights to resources that your organization owns. Make sure you have all the people necessary to make the application work available during the first few days of testing as a minimum. For example, if your application requires database access, ensure that a DBA is available to help create the required access levels. Coordinating the effort is a good job for DevOps, assuming your organization has someone assigned to this task. Don’t forget to include management staff as needed.

Performing Required Test Monitoring

Monitoring the testing company is an essential part of ensuring the tests go as planned. You need to ensure that the testing company performs all required tests (and doesn’t simply sign off on them), yet doesn’t perform any unauthorized (and potentially chargeable) testing. It also pays to monitor resources during testing. For example, the DBA should ensure that the testing process doesn’t damage any database data, even if that data is used specifically for testing purposes. Some application failures aren’t immediately apparent unless you perform the required monitoring.

Handling Unexpected Testing Issues

Testing is an antagonistic process. After all, you’re trying to force an application to fail. Any antagonistic process is going to produce unexpected results at times. You can spend as much time as you like stating that a given input should provide a given output, but the emphasis of that statement is on the word should. The reason you perform testing is to find those times when the results are different than you expected.

Of course, the testers handle most of these unexpected results using whatever system you have in place for reporting bugs. However, some unexpected results come from external sources, such as miscommunication between test team elements. In this case, reporting the issue as a bug won’t help because the problem isn’t within the application—it’s the result of some external source acting on the application or its testing process. To handle this sort of testing issue, you need a process in place that works outside the bug reporting system.

A common way of addressing these sorts of issues is to hold an impromptu meeting. The problem is that meetings can become unfocused, are hard to schedule, and may not produce any usable results because stakeholders need time to research issues. Some organizations rely on private forums or wikis to address these sorts of issues. In fact, products exist for handling this issue in all sorts of ways. You can see a list of 25 common developer collaboration tools at https://blog.profitbricks.com/top-25-collaboration-tools-for-developers/.

Discussing the Report Output with the Third Party

The reports that describe application performance during testing can vary and the parties involved vary as well. For example, most applications today rely on third-party libraries, APIs, and microservices. You may choose to report issues with these third-party elements to the vendor that created them as a means of making your application work better. Of course, you don’t want to share the entire report, including the reports on your personal code, with these third-party vendors. A third-party tester can address the need to contact other third parties about their code, but you need to have some policy in place for managing this issue. If you have the third-party tester perform the work, the third-party stakeholder can simply upload new versions of their code to the tester—greatly simplifying your role in the task. (You still need to know about any changes to third-party code because these changes could affect other applications.)

Reports that the tester generates need to address the various audiences in your organization. A management team may only want to see an overview that provides status information and the big numbers. Users may only want to know about issues that affect usability and the sort of interface the application presents. Developers want detailed reports that specify precise error locations and potential fixes for them. A DBA may only want to know the specifics of issues that affect the database. In short, reports come in various forms to address the needs of specific audiences.

Presenting the Report to the Organization

The application testing process generates the vast majority of the reports you need automatically, as part of running the test. A routing mechanism can ensure everyone who needs a report gets one through some method, such as email. However, getting a report to a stakeholder isn’t quite the same as getting the stakeholder to read it. Of course, the question is why anyone should care about this issue given that the stakeholder should take responsibility for reading the material. The problem is that stakeholders tend to ignore information until they really do need it, which is usually too late when it comes to application development. Stakeholders need to remain engaged in order to ensure that the application performs as expected when testing completes.

Depending on the development model you use, you may need to present reports to the organization as often as weekly. Some models actually call for a Monday meeting. Each Monday you present a report of the project process, any issues, and then follow it with a demonstration. Stakeholders who can see progress each week tend to become more involved in the project and provide better input, which means you get fewer questions about adding features at the end of the process when it isn’t possible to add them any longer.

Using Monday as the presentation day also gives the development, testing, and other teams time to react to suggestions or needs for application development. Instead of trying to rush a fix on Sunday, the various teams have the whole week in order to produce an updated application version that will amaze the stakeholders on Monday. As a result, the various teams actually get some rest and are able to do a better job.

Acting on Testing Recommendations

The result of testing, reports, discussion, and so on is that the various teams need to fix problems found by the various testers. Some problems are more important than others are and are most likely to cause problems after application deployment, so you work on these issues first. Part of the testing team and stakeholder interaction is to assign values to each problem that defines the importance of that problem. For example, you wouldn’t want to fix a low-priority fit and finish issue before you fix that showstopper issue that causes the application to crash on nearly everyone’s machine.

However, just fixing the problem doesn’t get the job done. In some cases, a problem has more than one answer and you might choose to create versions of the application that implement each answer to determine which one is best. The various stakeholders need to review the potential fixes and decide which one works best for them (which may not be the same fix in all cases). Creating and exercising all the potential useful fixes for a problem is part of acting on testing recommendations.

It’s also important to have the fix tested by the testers. However, you might decide not to have the fix tested immediately. Perhaps an interaction makes it hard or impossible to test the fix without repairing other problems in the application first. It’s essential to look at the dependencies of a particular issue and be sure to address the dependencies as well as the original problem.