Avoiding Useless Reports

Organizations and even individuals generate a plethora of reports in a lifetime, only a few of which are actually useful. A report is a kind of tool, so you must have a need for it to perform useful work. Some people view reports as informational, but this isn’t the case. If you can’t act on the content of the report, the information is useless and you may as well not take the time to read it. In fact, useless information is a significant contributor to the phenomenon of information overload.

Reports that contain useful information help you perform specific tasks. They provide the basis on which to act. With this in mind, reports that help you enhance the security of your application usually contain topics like these:

• Bugs

• Speed

• Reliability

• Usage

• Data access

The kind of report determines how you act on the information it contains. When you find that you don’t perform any actions in response to a report, the report is either faulty and requires change, or you really don’t need it. The information in reports generally inspires some sort of action for the following reasons:

Change in pattern

Data consists of patterns. You can see ebb and flow in the way data presents itself. In fact, whole branches of math are now devoted to the study of data patterns. When you see a change in a data pattern, even if that change doesn’t appear to present a problem, you need to investigate its source. Some patterns present reliability issues and others security issues. Of course, a change in pattern could simply come from a change in usage or some other factor, but you need to know why the pattern has changed.

Change in status

A change in status can mean many things. However, status indicators tell you that the ongoing flow of information has changed in some way, which means that you need to check for potential interface, reliability, security, or resource problems. Interface issues can result in security problems down the road and resource problems usually end up causing reliability problems.

Anomalous event

Nature constantly presents the potential to provide an anomalous event. For example, the glitch caused by lightning can present itself as an anomalous information event. However, anomalous events also occur when hackers probe a site looking for potential weaknesses. Some hacks can manifest themselves in as few as three anomalous events, so it pays to track down the source of each event.

Unexpected values

When a report contains information outside the expected range, you need to consider why the application generated such information. The unexpected value could come from a bug or it might be the result of hacker probes. Perhaps it’s the result of resource limitations on the server or bad data in the database. The fact is that unexpected values occur regularly and people simply dismiss them instead of acting upon them. Always verify the reasons for unexpected values when you can.

Keeping your desk clear of information you can’t act upon is an essential part of maintaining a secure application. When you clear away the clutter, you can begin to focus on the information that you truly need in order to locate and verify security issues in your organization. The key is to ask how you plan to act on the information contained in a report before you create a new one. In addition, getting rid of old reports that no longer serve a purpose is essential to eventually gaining control over the information flow in any organization.

Timing Reports to Upgrades and Updates

From a security and reliability perspective, you need good information to act on application changes that simply don’t work out. In order to obtain the information you need, it’s important to time reports to those times when you make upgrades and updates to the application. Timing may be as simple as changing when the system outputs the report or modifying report frequency so that you get the information a bit more often.

The flow of information should increase in proportion to the significance of the changes you make. However, as mentioned in “Avoiding Useless Reports”, you also need to avoid information overload. It’s when you become overloaded that you tend to miss patterns and critical events presented by the data. It’s important to maintain a balance between the information you really can use and the information you really need. When you start finding that reports go for days without initiating any action, then you know you’re experiencing information overload and need to reduce the information flow in some manner.

Increases in information flow due to upgrades and updates are temporary. You don’t need to continue receiving the additional information beyond a reasonable timeframe. Of course, the problem is determining when to reduce the information flow. Normally, you can start reducing information flow when the frequency of bugs and odd events decreases, and the patterns of application usage return to normal. Using this approach means that you can then divert some of your attention to other matters, such as planning the next update or upgrade.

Using Automatically Generated Reports

Monitoring software and other support tools usually provide you with some number of reports. The vendor creating the software usually develops these reports in response to user requests, so the reports often reflect precisely the data you need to perform general application monitoring. It’s important to go through the list of automatically generated reports and select just the reports that meet your needs. Selecting all of the reports and hoping you find the information you need is a bad idea because it leads to information overload and missed opportunities to correct application issues.

The generated reports will provide generic information, but sometimes you can make it more specific to your needs by selecting report options (when available). In many cases, the vendor will provide the means to customize a report so that it provides just the information you actually need, rather than a lot of information you don’t. The important thing is to run the report once or twice with all of the features in place so that you can see what’s available and determine how you can act on it. Once you know these two bits of information, you can start customizing the report to make it smaller and more efficient to use.

Using Custom Reports

The generic reports that the vendor provides with the product are a great way to get started monitoring your application for problems, but they usually don’t provide enough information about your specific needs. Of course, the vendor doesn’t know anything about your specific needs. The only way to obtain information that directly relates to how your specific application works in your environment and with your users is to create a custom report. When the monitoring software you use doesn’t provide the custom report creation capability you require, then it’s time to write an application that will obtain the information and format it in a manner that works. This section is about the kind of report that you create completely by hand (see “Using Automatically Generated Reports” for a discussion of customized reports).

Of course, some organizations go quite crazy with reports and have so many that no one really knows what they’re all supposed to do. Exercising restraint when it comes to custom reports is essential. Otherwise, you find yourself buried in information you don’t want, need, or understand and also have the extra burden of maintaining the report generation software. The following sections describe methods you can use to make your custom reports better reflect your needs and generate just the information you require to keep your application safe.

Creating Consistent Reports

An issue that is hard to overcome—but important to contemplate—is consistency. When your reports format data in several different ways and present it in different orders, it can cause confusion. Anyone using the report is bound to make mistakes when glancing at the report, and might end up assuming the data means one thing when it says another. The reports you use and create should consider the following consistency issues:

• Data appears in the same format.

• Data ordering is the same whenever possible.

• The report doesn’t reference data using multiple, conflicting approaches.

• Any data cleaning and filtering works in a consistent manner to produce the same information for each report (based on the use of the same algorithm).

• Graphically presented data mirrors the textual data.

When you update reports, make sure you poll users for consistency issues. The people who rely on the information to perform real-world tasks will likely notice discrepancies faster than someone who simply manages, maintains, or outputs the report. Some data can be quite nuanced, making it hard to decipher potential problems with its appearance and layout.

Using Reports to Perform Specific Application Tasks

As mentioned quite a few times in this chapter so far, a report is only useful when it provides an actionable output. However, some reports may not have human readers in mind. It’s possible to create reports that enhance application automation. Monitoring and management applications can read log files and use the data they contain to determine when to perform application management tasks. You don’t want the maintenance actions to happen all the time or on a particular schedule—it’s more important to perform the maintenance only when necessary in order to reduce resource usage and improve application availability. Because other software reads and acts upon the information in these reports automatically, you need to create them with an eye toward consistency and accuracy. The application doesn’t care whether the report is beautiful or not, but it does care that the data it contains reflects the true state of the application.

Determining Which Data Sources to Use

Internal reports tend to work with organization-specific data. These reports reflect the precise nature of the applications running on your systems and within the environments that you provide. You can see the usage patterns of your users and anomalous events that are specific to your system. In other words, internal reports tend to focus on the data that a vendor can’t provide as part of a generic report.

The problem with internal reports is that everyone seems to assume that everyone else is trustworthy and that the data included in the report will never leave the organization. It’s important to realize that many data breaches that occur today happen because a trustworthy employee left their laptop at the airport or someone with fewer scruples sold it to the highest bidder. With this in mind, you need to consider that any data you make available as part of a report will eventually see the light of day in the outside world, whether you want it to appear there or not. The following list provides you with some issues to consider when using internal data sources for a report:

• Ensure the people using the report actually require the data you’re including.

• Define protocols, with penalties, for working with sensitive data.

• Create a plan for dealing with data breaches that involve sensitive data (procedures you intend to perform beyond the normal data breach protocols).

• Track the users of sensitive data to make it easier to determine where the source of a data breach occurred.

• Secure the sensitive data and keep it separate from less sensitive data.

Protecting your data is paramount today. Too many organizations are appearing on the front page of the trade press or in the major news media after making a seemingly small mistake that ends up giving hackers access to sensitive information. In many cases, the fault lies with incorrectly using an internal data source that an organization is supposed to jealously guard by making it available to people who really don’t need it. The best question you can ask yourself when creating a new report is whether the persons who will view that report actually need to see the data you’re providing.

Specifying Report Uses

Internal reports can and do use sensitive data. If you were to lock up every potentially sensitive piece of information, the organization could never function properly. There is a time and place for making the sensitive data that an organization holds known. However, you still want to reduce the security risks associated with sensitive data by ensuring that you define the uses for reports that you generate and spell out how users are to interact with them. An organization requires an enforceable set of rules for dealing with sensitive data contained in reports.

It’s important to consider precisely how the report uses the data. Extrapolations and data science techniques can make sensitive data even more damaging when released in the wrong venue. Then again, you can use data cleaning and filtering techniques to make the data less sensitive and possibly acceptable for use in presentations to the organization as a whole. For example, data associated with individuals is more sensitive than disassociated data. Cleaning techniques would allow you to retain the statistical value of the data without associating it with specific individuals.

Reports should also have documentation associated with them. You need to know who created the report and why. The report should have a review date to ensure that it continues to have a pertinent purpose in the organization. As part of the documentation, you need to define some elements of report security:

• Detail who can access and read the report.

• Specify who can generate the report.

• Define how users can view the report and where (some reports may not be suitable for download to a mobile device, for example).

• Create a process for ensuring that all copies of the report are destroyed at some point (except an archival copy maintained in a secure location).

• Provide contact information so that anyone with questions knows who to ask about the report and its content.

Obtaining Completed Reports from Third Parties

Any sort of third-party support product you obtain, such as application or security testing, needs an associated report. The report must detail the service rendered and when. It should tell anyone viewing it why the third party performed the service and the results of performing the service (for example, testing should detail when the tests succeeded or failed and provide the means to look at individual results). Any reports you require need to appear as part of a deliverables list that the third party must complete prior to receiving payment or it’s likely that you won’t receive all the reports you need.

As with internally created reports, any report you get from a third party should provide an impetus for action. Third parties tend to generate reports that are meant to impress, rather than inform. The glitzy paper and persuasive wording won’t help much when you’re dealing with a security breach possibly caused by an error on the part of the third party or a lack of attention to detail. In short, you need to view the report in the same way that you do internal reports—always questioning the value the report provides to you and your organization.

Whenever possible, work with the third party to generate reports that specifically address the needs of your organization, rather than accepting generic reports that the third party creates for everyone. The report should reflect your application environment, use of resources, personnel, and other factors that make your organization unique. The custom report will probably cost more, but you obtain more information from it and the savings in employee time often produce a return in excess of the additional cost.

Developing Reports from Raw Data

Some third parties will provide an API you can use to access the raw data associated with your application. Rather than trust the third party reports completely, you can use the raw data to generate your own reports. The service normally costs more and you won’t get it with all third-party providers, but it does pay to look for it. When you do create custom reports, use the guidelines in “Using Custom Reports” to create them. The important issue is to ensure you get just the information you need, in the form you need it, in order to perform necessary tasks.

Keeping Internal Data Secure

A third party may have access to some internal data for your organization. The data may not be sensitive, but you still want to protect it because you know that a third party has access to it. When working with a third party, you want to treat all data the third party can access as sensitive data, following the rules found in “Creating Internal Reports”. In addition to following these rules, you want to know as much as you can about the third party. Trusting the third party comes only after you perform the proper steps to verify you truly can trust them. Here are some suggestions for ensuring that your application, its data, and your organization’s data remain secure when working with a third party:

• Verify the identity of any third party who will work with the data.

• Perform a background check to ensure the third party is trustworthy.

• Track any access the third party makes to shared data and ensure you follow up immediately on any anomalies you experience.

• Ensure the third party signs a nondisclosure agreement (NDA).

• Monitor the third party to ensure there aren’t any attempts to access sensitive data.

• Create a procedure for handling third-party data breaches or managing an issue regarding compliance with security protocols.

Obtaining User Feedback

User feedback is a kind of report and it’s an essential part of maintaining a secure environment for your application. At one level, users who are dissatisfied with the performance of an application and believe that their requests for service will go unanswered tend to find solutions to the problems on their own—usually with the resulting security breaches and reliability issues. At another level, user feedback tells you how your application is actually serving user needs. In many cases, users require help and service that they can’t verbalize, partly because they don’t know the application isn’t working as you had intended for it to function.

It’s possible to obtain user feedback in all sorts of ways. Some methods work better than others do at obtaining specific kinds of feedback. For example, a comment form will tell you how a user feels about the application, but it won’t tell you how a user is actually interacting with the application because the user input is biased (more on this issue in “Determining the Usability of User Feedback”). When working with a group of users, you often have to derive a middle position based on the varying inputs you receive. Each user may feel that his or her own input is unbiased, but everyone presents bias when expressing an opinion. The bias affects the input you receive, which affects its usability. The following list provides you with some ideas for obtaining user feedback, but it’s important to remember that some of these methods are invasive and others tend to produce biased results:

• Create feedback forms that the user can send to you to request new features or simply comment on existing features.

• Sponsor group discussions where a number of users can talk about the relative merits of the application and define what they would like to see changed.

• Log the user’s interactions with the application by adding checks to the software.

• Monitor user interactions with the application platform so that you can better understand how the user works within the application environment.

• Track user access outside the application environment, such as through access to a database, API, or microservice.

• Request user input whenever a special event (such as a crash due to a bug) occurs.

Determining the Usability of User Feedback

Obtaining user feedback is helpful, but only when you consider the circumstances under which you obtained the data. Talking directly with the user will always produce a biased result. Monitoring the user may produce an unbiased result, but if the user knows that the monitoring is taking place, his or her actions won’t reflect actual usage conditions and the data you receive will be suspect. Logging user interactions through software checks doesn’t provide the depth of information that monitoring provides. The point is that you must consider how you received the information before you can make any decisions about its usefulness in accomplishing a specific task.

To put this into perspective, a user might request a particular feature because it seems like an interesting thing to do. However, testing shows that the feature will likely increase the risk of a data breach because of the way in which it accesses data. In addition, monitoring shows that the user doesn’t receive much benefit from the feature—it’s rarely put to use in a real-world setting. Consequently, you have three bits of information you need to correlate in order to determine whether to keep the feature. Removing the feature will almost certainly elicit comments from the user, so you need a good reason to remove it. In situations such as this one, you need to consider what other sorts of usable user feedback might help. For example, you might bring up the issues with the feature in a group session and rely on the group’s input to help you understand the need for the feature.

Application security often hinges on the user’s participation. Social engineering attacks are devastating enough, but trying to thwart the efforts of a disgruntled employee can be even harder. Part of the purpose of user feedback is to ensure the user remains committed to working with the organization as a whole to keep the application and its data safe. In some cases, you need to weigh the effects of accepting user feedback that may not seem very helpful at the outset in order to maintain the user’s good feelings toward the application.

During the process of evaluating user feedback, you must also consider the politics of the organization. User feedback is always biased, but how biased often depends on the perceived value of the application to the user and the politics behind the decision-making process. Some situations require that you partially disregard the direct user feedback and try to obtain feedback through indirect means (such as monitoring) in order to determine where politics end and the true bias begins.

Make sure you consider all user feedback with the assistance of your development team and keep management informed about your progress. However, it’s important to remember that your development team and management are both made up of people with their own biases and political agendas. It’s human nature to disregard the user input you really didn’t want to hear in favor of the opinions of the development team that correspond closely with your own. However, sometimes the user really is correct and you need to consider that view before making decisions.

As part of the decision-making process, also consider the other information sources at your disposal. The reports you generate not only tell you about the application, platform, third-party services, and so on—they also tell you about the state of your organization and help you make better decisions about user feedback you receive. Don’t forget to consider the input from outside sources, such as trade press articles. A report that a certain type of hack is taking place might weigh heavily against including specific application features that would make it easier for a hacker to gain access to your application. In short, use all of your resources to get the big picture, rather than taking a myopic view of just the user input you receive.