Chapter 16. Vulnerability Management

Contrary to what some vendors’ marketing material would have us believe, a huge quantity of successful breaches do not occur because of complex 0-day vulnerabilities, lovingly handcrafted by artisanal exploit writers. Although this does happen, a lack of patching, failure to follow good practices for configuration, or neglecting to change default passwords are to blame for a far larger number of successful attacks against corporate environments. Even those capable of deploying tailor-made exploits against your infrastructure will prefer to make use of these types of vulnerabilities.

Vulnerability management is the terminology used to describe the overall program of activities that oversees vulnerability scanning and detection through to remediation. This is a program that ultimately raises the security of your network by removing potential flaws.

Vulnerability assessment is a different discipline than penetration testing, typically carried out by different people; however, the term is often used interchangeably by those who are not aware of the differences.

Unlike penetration testing, vulnerability assessment is automated or semiautomated, continuous, and less focused on bespoke systems and applications. Vulnerability assessment tools generally search for flaws such as missing patches, outdated software, common configuration errors, and default passwords. Vulnerability scans ideally operate on an ongoing basis, rather than a one-time or annual assessment.

Issues that are often discovered by a vulnerability assessment tend to be known issues found in widely distributed software, and so vulnerabilities in your own code are less likely to be discovered by a vulnerability scanner. That is the role of penetration testing and code analysis. Vulnerability scanners do not attempt to adapt to the environment; rather, they attempt to enumerate an environment, discovering which software is installed, which versions of software, what some of the configuration options are, and if any default accounts remain with the default password.

In this chapter we will discuss vulnerability scanning at a technology level, and how this can form part of a larger program designed to better manage vulnerabilities across your environment—and ultimately improve the overall security of your systems.

How Vulnerability Scanning Works

Of course the exact techniques used by a vulnerability scanner will vary from tool to tool, especially in an industry like information security where techniques fall in and out of favor fairly quickly.

In the simplest form, a vulnerability scanning tool will attempt to determine the information that it requires by probing the target and trying to solicit a response that will allow it to determine some detail about the software running on the host and its configuration. How this is achieved will vary depending on whether the type of scan is authenticated or unauthenticated.

Authenticated versus Unauthenticated Scans

Vulnerability scans can be both authenticated and unauthenticated; that is, operated using a set of known credentials for the target system or not.

This is because authenticated scans typically produce more accurate results with both fewer false positives and false negatives.

An authenticated scan can simply log in to the target host and perform actions such as querying internal databases for lists of installed software and patches, opening configuration files to read configuration details, and enumerating the list of local users. Once this information has been retrieved, it can look up the discovered software, for example, and correlate this against its internal database of known vulnerabilties. This lookup will yield a fairly high-quality list of potential defects, which may or may not be further verified before producing a report depending on the software in use and its configuration.

An unauthenticated scan, however, will most likely not have access to a helpful repository of data that details what is installed on a host and how it is configured. Therefore, an unauthenticated scan will attempt to discover the information that it requires through other means. It may perform a test such as connecting to a listening TCP socket for a daemon and determining the version of the software based on the banner that several servers display. This technique can be easily demonstrated against an SMTP server using telnet, for example:

$ telnet my.example.com 25
Trying 192.168.1.25...
Connected to my.example.com.
Escape character is '^]'.
220 my.example.com ESMTP Exim 4.82 Ubuntu Tue, 01 Jan 2016 00:00:00 +0100

In this example an unauthenticated scanner would typically assume that the SMTP server is running Exim version 4.82 on Ubuntu. The scanner would then compare this server to an internal database of vulnerable SMTP servers and output a report based on whether or not this version was listed to be susceptible to any vulnerabilities.

However, there is nothing to say that the administrator of the server isn’t really running a much older or vulnerable version of Exim, or any other mail server for that matter, and is just displaying a false banner via a configuration option in order to hamper this sort of profiling. This could have been achieved in Exim with this simple configuration option:

smtp_banner = "${primary_hostname} ESMTP Exim 4.82 Ubuntu $tod_full"

Ideally, the toolset would use the version in the banner as a best current guess, but conduct other tests to determine if that is the true version. For example, different mail servers and different versions of the same mail server may act differently with regard to supported features, the ordering of header information, or other nuances that can be used to fingerprint a system. This, however, is not always the case; it depends on the toolset. In lieu of tools that automatically complete these tasks, it often falls to the person reviewing the results to verify them, or to at the least make decisions with the knowledge that there is a chance of incorrect results being gathered this way.

As described earlier, an authenticated scanner will log in to the host and execute commands to determine the correct version of the software installed, not just make assumptions based on the banner.

Authenticated scans not only remove ambiguity from results of this nature, but can often highlight issues that are not discoverable from an unauthenticated scan. For example, a local privilege escalation vulnerability via an installed command-line tool would most likely not be visible by probing services that are listening for network connections via an unauthenticated scan. These sorts of vulnerabilities are still important to discover and remedy, and when combined with another exploit such as a remote code execution vulnerability, would create a remote root vulnerability. That is, vulnerability #1 is used to remotely execute a command as an unprivileged user, and that command executes an exploit against vulnerability #2 to escalate from an unprivileged user to a privileged user such as root in Unix and Administrator in Windows.

If authenticated scans can reduce both false positives and and false negatives, why is there even an option for unauthenticated scans? There are a couple of reasons.

While it is true that authenticated scans are more accurate, there is one risk that should be highlighted before running headlong into using them. They are authenticated, which means that they, by definition, will have access to some sort of authentication credential, be that username and password combination, ssh keys, or something else. These credentials need to be managed as well as any other credentials would be. It is therefore sensible to set up dedicated credentials just for the scanning server to use. If possible, schedule these credentials to only be active during pre-authorized scanning windows; that is, they should be disabled during times that scanning is not scheduled to take place to minimize the chance of abuse. If this is not possible, ensure that an audit log of login times and IP addresses is produced, and that this audit log is reviewed to ensure that login times and locations align with those of the scheduled scanning time.

The results of unauthenticated scans are what attackers would see, assuming that they do not already have credentials for your systems, and so even if not a completely true representation of vulnerabilities, unauthenticated scans can provide insight to what is available from an attacker’s perspective.

Warning

There have been instances of legacy applications and network equipment encountering performance issues, and in some instances crashing during a a simple port scan, nevermind a full vulnerability assessment. For this reason scans should take place during a prearranged engineering window, and under change control.

After a couple of successful scans it will become easier to make the case for regular scans to take place on a predetermined schedule.

Using some trial runs to place people at ease, we have managed to take organizations from being unable to run scans at all to having a fully automated scan that tests a different area of the network each day, recurring every week. That organization is now never more than a week out of date for vulnerability data regarding any host.

Vulnerability Assessment Tools

As with most other security disciplines, there are many tools available for vulnerability assessment, both free and commercial. The value that each tool brings is entirely dependent on your particular environment and the staff that will be operating the tools. When selecting a tool or tools to use, the key areas to consider are:

Coverage for your particular technology stack: Many vulnerability assessment tools will have gaps in coverage, especially for more esoteric systems. For example, in my case, one of the leading commercial solutions that had thorough coverage for Windows and Linux systems had a gaping hole when it came to AIX systems. A guess as to which system we were running in our environment is left as an exercise for the reader. Some tools are aimed at very specific areas, such as web applications. While useful in their specific area due to the focus, such tools often tend to not give much visibility into operating system issues, and should be used in tandem with other tools to provide the appropriate coverage.
Automation: Some tools are heavily automated, run on a schedule, self update, and can effectively be left to run with only minimal maintenance on a day-to-day basis, producing reports as scheduled. Others are almost guided penetration-testing tools that require specific technical knowledge to obtain reasonable results. Picking the right tool based on experience, technical knowledge, and how much time you have to operate the tool is essential. Most organizations tend to lean toward a general-use tool for vulnerability assessments, and favor regular penetration tests, code review, and code analysis to find weaknesses in these areas.
Scope: Vulnerability scanners come with a wide range of features, and it is worth determining what the scope of your vulnerability assessments will be when assessing possible tools. For example, some will focus on missing operating system patches, while others will have more options with regard to web application scanning. Some will use auto discovery, while others will need to be told which IP addresses are in vulnerability assessment. You can make more informed decisions regarding which tool is most appropriate for you.

Note

Additional results can be gained by using the guided penetration test–type tools, if used by an experienced operator. A better use of time, however, is probably to use automated tools to capture low-hanging fruit such as missing operating system patches and older versions of software, and undertake a full penetration test via an external company to uncover issues in bespoke applications.

Vulnerability Management Program

The vulnerability management program is not only a matter of technology, it is comprised of the policies, processes, and procedures that are used to discover vulnerabilities, and see them through to remediation. After all, what is the point in discovering flaws in your system if you are not going to fix them?

Working on the assumption that if you are reading this, you probably do not have a vulnerability management program in place at this point, we are going to need to catch you up before we move on to business as usual.

Program Initialization

If you have existing infrastructure but no vulnerability management program in place, it is incredibly likely that when you run your first scan you are going to end up reading a very long report. This can be a daunting experience filled with red lines on graphs. Fret not—with some pragmatic planning and prioritization this is a manageable task.

Let’s assume that the results of your first scan have highlighted vulnerabilities across a wide range of hosts, with a wide range of criticality ratings from low to critical, and that you probably have a mixture of different operating systems in the mix. The normal methods of prioritization, outlined in “Business as Usual”, may not be practical. If you can skip directly to the business-as-usual process, you should, as it permits the use of better prioritization techniques.

Let’s break down the vulnerabilities in the report by the operations team who will most likely have the access to systems required, and the appropriate job responsibilities to deploy fixes. Typically this is either by technology type (Linux team, Windows team, network team, etc.) or by function (finance servers, IT servers, human resources servers, etc.). Now you have a separate list of fixes for each team to implement.

Next let’s try to create batches of multiple fixes or instances of fixes that can be deployed during a single change or engineering window to maximize the number of fixes that can be deployed at any one time. The easiest two approaches are:

Look for vulnerabilities that occur frequently throughout your environment. By patching the same vulnerability or making the same configuration change over and over across the entire environment, certain tasks such as pre-implementation testing can be performed only once, reducing the amount of time needed to roll out the fix. The remediation steps are likely to be repeated identically on each host that it is deployed to, leading to a predictable outcome on each host and less friction for those who are making the changes.
Pick a particular host of group or hosts and attempt to address all, or at least many, patches on that host or group. Although the range of patches is more varied than in the previous option, there is only one host or group of hosts to raise change requests for, manage outages for, and to monitor throughout the process. This will result in fewer reboots because, hopefully, a single reboot can be used to roll in multiple patches, and you may only have one team that needs to work to deploy all the patches.

Once the patches are grouped using whichever system works best for your particular organization, you should prioritize (see “Remediation Prioritization”) the batches of remediation work to ensure that the most important changes are implemented as quickly as possible.

By using one of these approaches, it is possible to deploy large numbers of patches during an engineering window. By repeating this process regularly, it should be possible to catch up to a point whereby you can move to the business-as-usual phase.

Business as Usual

Unlike the program initiation phase, which is often predominantly comprised of “catching up” remediation activities in bulk patching, config change, or upgrade cycles, operating vulnerability management as a business-as-usual process relies upon a more systematic approach to handling vulnerabilities. The process should look something like:

Discover vulnerabilities within your environment. This is typically via a combination of scheduled automated scans, vendor announcements, and scheduled releases.
Prioritize remediation activities in order to determine the priority order and timeline to remedy each vulnerability.
Assign remediation activities as appropriate. In many organizations there may be separate teams to manage different systems.
Track and monitor remediation activities. This will often happen through a check of change control tickets and ensuring that items are not present during the next scheduled vulnerability scan, indicating that they have been fixed successfully.

Remediation Prioritization

It is all very well having a long list of vulnerabilities that require remediation, but the reality is that for most people the ability to remediate everything quickly is purely aspirational. Not only is there the issue of time and resources, to carry out work, but also problems with obtaining a maintenance window, raising change control tickets, patch testing, and all manner of other potential issues. For us mere mortals there is a need to prioritize the work in order to ensure that the more important vulnerabilities are addressed in the most timely manner, while less important issues can wait a little longer. This brings us to the important factor in prioritization: what is “important”?

Typically this is the point in the book where you will find a 3x3 or 5x5 matrix with differing levels of panic along each axis to tell you how to deal with your vulnerabilities. Ultimately what these diagrams come down to is having a repeatable process to determine how quickly a vulnerability should be remediated that works for your organization.

Nearly every system of vulnerability prioritization, matrix or otherwise, will use the severity rating placed on the vulnerability as one of the metrics. The severity rating is often derived from a combination of the impact of successful exploitation without any context of your environment, and the “likelihood” of exploitation. Likelihood is typically comprised of factors such as complexity, preconditions, if user interaction is required, and other items that could influence the likelihood of a successful attack.

Ultimately, all these methods of determining priority are based on determining when each item should be patched. Use what works for and is relevant to your organization.

Severity Ratings

There are multiple systems used to calculate severity ratings. The most common is probably the Common Vulnerability Scoring System, or CVSS. However, some vendors, such as Microsoft, will produce their own severity ratings.

Ratings

Unless you are someone who is versed in risk and risk language, or you have a particular interest in the field, it is probably advisable to accept the vendor-supplied rating and continue. The need to set a completely accurate severity rating is not always present. For the purposes of this exercise, the severity rating serves to provide approximate categorization.

As alluded to earlier, the missing element from the vendor-supplied severity rating is context. A vulnerability present on a standalone PC with no sensitive data and no network connection is going to have a very different potential impact to the same vulnerability discovered on an internet-facing web server; context makes all the difference. If your infrastructure is suitably small you may be able to prioritize based on vendor rating alone, but context can provide further granularity in order to prioritize more effectively. This context typically forms, in one way or another, the other axis of the aforementioned matrix.

There are multiple ways that you could add context, and relevance will depend on your organization. Some examples are:

Sensitivity: The more sensitive the data held on the device, the higher the priority. This is mostly driven by breach-type scenarios whereby loss of personal or financial data is a worst-case scenario both from a regulatory and PR point of view.
Volume of hosts: If all hosts are even, then this approach can work well, as a vulnerability affecting 300 hosts will be more important than one that affects only 5 hosts. In reality, however, most hosts are not created equal.
Exposure: That is, how likely it is that a host be exposed to an attacker. For example, an internet-facing host will probably have a higher exposure than one on an internal network.

Having determined how you are going to rate each vulnerability, the final step is to set timelines for remediation to ensure a consistent approach. For example, an internet-facing, critical vulnerability should be remediated within 1 day, while a low criticality rating affecting only desktops could wait 30 days. Mapping out your strategy with timelines brings consistency, which in turn makes it easy to set expectations and to track progress of remediation activities. Meeting the timeline could be part of internal SLAs or KPIs for teams and is a relatively easy metric to track.

An example, which should of course be modified to your own needs, could be Table 16-1.

Table 16-1. Vulnerability remediation time table
	Isolated LAN	Internal LAN	Partner Facing	Internet Facing
Critical	7 days	2 days	1 day	1 day
High	7 days	5 days	3 days	3 days
Medium	14 days	7 days	5 days	5 days
Low	21 days	7 days	14 days	14 days

Risk Acceptance

In the case that a vulnerability cannot be mitigated, either within the agreed timelines or at all, there is a process known as risk acceptance.

Risk Acceptance

Security professionals wince, quite rightly, at the mention of risk acceptance. Although this may seem quite obvious, experience has borne out that this bears repeating, quite possibly twice.

When a risk is “accepted” via a process, it does not, in reality, disappear. Software is still vulnerable, the risk is still present on the system, and an attacker can still make use of it. Merely, a suitably senior member of staff has acknowledged the existence of the vulnerability and accepted responsibility for the decision to permit its continued existence in the environment.

This is only a “fix” on paper.

(Read this just one more time.)

Risk acceptance is, as it sounds, the process of a member of staff of predetermined seniority being able to document acceptance of a risk. That is, he agrees that for whatever reason, the risk is permitted to remain within the environment and that he accepts responsibility for this decision. Typically this is for reasons such as software interdependence, which means that remediating a vulnerability will cause some sort of other issue, or that upgrades require the procurement of a license, for example.

Risk acceptance should be used as a last possible measure, and all acceptances should have an expiry placed on them and not remain in perpetuity. This helps to ensure that risks are reviewed, revisited, and hopefully remediated instead of living on forever in an accepted and perpetually renewed-acceptance state. Even a vulnerability that is expected to remain for a long period of time should only have a short acceptance period, which is reviewed and reissued upon expiry if needed. This ensures that its continued existence is a cognizant decision, with a named individual responsible for that decision.

Conclusion

A vulnerability management program allows you to assess, log, and remediate vulnerabilities within your environment using a largely automated set of processes and tools. By following even a simple such program, the issues responsible for a large number of breaches, unpatched systems, and simple configuration errors will be drastically reduced.