Chapter 7. Data and Sensors in the Active Domain

In this chapter, we will discuss the active domain. Active domain data refers to data that the analyst specifically initiates the generation and collection of. Active data will involve the use of probing tools (e.g., ping, traceroute) and scanners (e.g., nmap). It is polled and triggered; as opposed to the continuous collection of the network and service domains, active analysis is done as a specific event or in response to specific anomalies.

Active work is client-focused, informative, and expensive. The analyst takes on the perspective of a client of an application (or, alternatively, an attacker) and is able to gather a lot of information about what a host does, although not necessarily much about what the host is for. But this fine-grained information doesn’t come cheaply; intensive scanning is expensive, it interferes with the target, and it can take an enormous amount of resources to process all the data received.

Chapters related to this one include Chapter 2, on the basics of network traffic, Chapter 13, which discusses the observation of fumbling, and Chapter 17, which is about the use of threat intelligence data.

Discovery, Assessment, and Maintenance

The foundation of operational information security is inventory management. The less you know about the assets you’re protecting—both what they are and what they’re for—the more any security plan is filled with guesswork. Some amount of guesswork is inevitable; networks change minute by minute as wireless devices appear and disappear, users reconfigure hosts, and the workday progresses. However, a well-executed network monitoring strategy will manage this guesswork and point out how to reduce it.

It’s in that context that we consider the interaction of active domain information and the information we’ve discussed elsewhere in this book. Active domain data is too expensive to collect universally and constantly, and the issues of vantage affect active data collection profoundly. Effective network monitoring will use network data to guide active collection. I’ll be referring throughout this chapter to three key actions: discovery, assessment, and maintenance.

In the context of this book, discovery refers to identifying the existence of assets within a network. Discovery can range from knowing that a particular IP address is active, to a complete inventory of the asset. Assessment, after discovery, applies security knowledge to that information to determine risks and courses of action. Assessment depends heavily on expertise, and with many commercial assessment products you’re paying primarily for that expertise. Maintenance is the ongoing process of checking an asset to ensure that it hasn’t changed. In the conclusion of this chapter, I will discuss how to combine active and passive monitoring to conduct an inventory.

Discovery: ping, traceroute, netcat, and Half of nmap

Discovery is the process of discovering and inventorying assets on your network. Discovery is progressive, and I can break it into two broad steps: determining something is there, and then determining what that something is.

The first step begins with a list of IP addresses, and involves determining which addresses within your network are dark (routable, but no host) and which addresses are lit. The basic tools for doing so are ping, traceroute, and nc. Once you’ve determined something is there, you have to figure out what that something is—this generally involves working with nmap and a collection of service clients built with nc and other tools to figure out what services are visible on the host.

Checking Connectivity: Using ping to Connect to an Address

Given an IP address, the most basic command-line tool for determining something is present is ping. ping works by using ICMP (see “Packet and Frame Formats”) messages. ping sends an ICMP echo request (type 8, code 0) to the target. On receiving an echo request message, the target should respond with an echo reply (type 0, code 0). Example 7-1 shows sample output of ping and a pcap of the contents.

Example 7-1. ping output

$ ping -c 1 nytimes.com
PING nytimes.com (170.149.168.130): 56 data bytes
64 bytes from 170.149.168.130: icmp_seq=0 ttl=252 time=29.388 ms

$ tcpdump -Xnr ping.pcap
reading from file ping.pcap, link-type EN10MB (Ethernet)
20:38:09.074960 IP 192.168.1.12 > 170.149.168.130:
        ICMP echo request, id 44854, seq 0, length 64
    0x0000:  4500 0054 0942 0000 4001 5c9b c0a8 010c  E..T.B..@.\.....
    0x0010:  aa95 a882 0800 0fb8 af36 0000 5175 d7f1  .........6..Qu..
    0x0020:  0001 24a6 0809 0a0b 0c0d 0e0f 1011 1213  ..$.............
    0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
    0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
    0x0050:  3435 3637                                4567
20:38:09.104250 IP 170.149.168.130 > 192.168.1.12:
        ICMP echo reply, id 44854, seq 0, length 64
    0x0000:  4500 0054 0942 0000 fc01 a09a aa95 a882  E..T.B..........
    0x0010:  c0a8 010c 0000 17b8 af36 0000 5175 d7f1  .........6..Qu..
    0x0020:  0001 24a6 0809 0a0b 0c0d 0e0f 1011 1213  ..$.............
    0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
    0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
    0x0050:  3435 3637                                4567

Note first the size of the packet and the ttl value. These values are usually set by default by the TCP stack. In the case of macOS, the ICMP packet has a 56-byte payload, which results in an 84-byte packet (20 bytes of IP header, 8 bytes of ICMP header, and 56 bytes of payload). The type and code are at 0x0014–0x0015 (08 for the request, 00 for the response). After the ICMP header, note that the contents of the packet are echoed. ICMP has a concept of a session, and in many cases, messages are sent in response to packets from entirely different protocols. Different ICMP messages use different techniques to indicate their point of origin; in the case of ping, this is done by echoing the packet’s original contents.

ping is a simple application: it sends an echo request with an embedded sequence identifier. The application then waits until a specified timeout (usually on the order of 4,000 ms); if the response is received in that time, the response is printed and the next packet is sent. ping is a diagnostic tool, and any serious implementation will provide a number of command-line switches for manipulating packet composition.

Sweeping Pings and Ping Sweeping

These are actually different terms, although Google gets confused when you enter a search for them. A ping sweep (or ping sweeping) is a scanning technique that systematically pings all the IP addresses assigned to a network to determine which ones are present and which ones are not. Ping sweeping is supported by nmap and a number of other scanning tools, although you can write a script to do it in about 20 seconds.

A sweeping ping, in contrast, is a sequence of ping messages that undergo size increases with each packet. Sweeping pings are intended to diagnose channels by identifying traffic manipulation or MTU issues. Sweeping pings are enabled by a command-line option on most modern ping implementations.

It’s not uncommon to find networks blocking ICMP messages. Ping sweeping is consequently a middling tool for finding hosts on a network; direct TCP or UDP scanning will generally be more effective.

Tracerouting

traceroute is a tool and technique to identify the routers that forward packets from point A to point B. traceroute produces a sequential list of routers by manipulating packet TTLs.

The TTL (time-to-live) field of an IP packet is a mechanism developed to prevent packets from bouncing through the internet forever. Every time a packet is forwarded by a router, its TTL value decreases by one. When the TTL reaches zero, the forwarding router drops the packet and sends an ICMP time exceeded (type 11) message to the source IP address of the dropped packet.

Here is an example of a traceroute command in action:

$ traceroute www.nytimes.com
traceroute to www.nytimes.com (170.149.168.130), 64 hops max, 52 byte packets
 1  wireless_broadband_router (192.168.1.1)  1.189 ms  0.544 ms  0.802 ms
 2  l100.washdc-vfttp-47.verizon-gni.net (96.255.98.1)  2.157 ms  1.401 ms
    1.451 ms
 3  g0-13-2-7.washdc-lcr-22.verizon-gni.net (130.81.59.154)  3.768 ms  3.751 ms
    3.985 ms
 4  ae5-0.res-bb-rtr1.verizon-gni.net (130.81.209.222)  2.029 ms  2.314 ms
    2.314 ms
 5  0.xe-3-1-1.br1.iad8.alter.net (152.63.37.141)  2.731 ms  2.759 ms  2.781 ms
 6  xe-2-1-0.er2.iad10.us.above.net (64.125.13.173)  3.313 ms  3.706 ms  3.970 ms
 7  xe-4-1-0.cr2.dca2.us.above.net (64.125.29.214)  3.741 ms  3.668 ms
    xe-3-0-0.cr2.dca2.us.above.net (64.125.26.241)  4.638 ms
 8  xe-1-0-0.cr1.dca2.us.above.net (64.125.28.249)  3.677 ms
    xe-7-2-0.cr1.dca2.us.above.net (64.125.26.41)  3.744 ms
    xe-1-0-0.cr1.dca2.us.above.net (64.125.28.249)  4.496 ms
 9  xe-3-2-0.cr1.lga5.us.above.net (64.125.26.102)  24.637 ms
    xe-2-2-0.cr1.lga5.us.above.net (64.125.26.98)  10.293 ms  9.679 ms
10  xe-2-2-0.mpr1.ewr1.us.above.net (64.125.27.133)  20.660 ms  10.043 ms
    10.004 ms
11  xe-0-0-0.mpr1.ewr4.us.above.net (64.125.25.246)  15.881 ms  16.848 ms
    16.070 ms
12  64.125.173.70.t01646-03.above.net (64.125.173.70)  30.177 ms  29.339 ms
    31.793 ms

As the next code block shows, traceroute sends an initial 52-byte message, and then proceeds to receive sequential information about each address it contacts en route to 170.149.168.130. Let’s look at the payload in more depth:

$ tcpdump -nXr traceroute.pcap  | more
21:06:51.202439 IP 192.168.1.12.46950 > 170.149.168.130.33435: UDP, length 24
        0x0000:  4500 0034 b767 0000 0111 ed85 c0a8 010c  E..4.g..........
        0x0010:  aa95 a882 b766 829b 0020 b0df 0000 0000  .....f..........
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000                                ....
21:06:51.203481 IP 192.168.1.1 > 192.168.1.12: ICMP time exceeded in-transit,
        length 60
        0x0000:  45c0 0050 a201 0000 4001 548e c0a8 0101  E..P....@.T.....
        0x0010:  c0a8 010c 0b00 09fe 0000 0000 4500 0034  ............E..4
        0x0020:  b767 0000 0111 ed85 c0a8 010c aa95 a882  .g..............
        0x0030:  b766 829b 0020 b0df 0000 0000 0000 0000  .f..............
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
21:06:51.203691 IP 192.168.1.12.46950 > 170.149.168.130.33436: UDP, length 24
        0x0000:  4500 0034 b768 0000 0111 ed84 c0a8 010c  E..4.h..........
        0x0010:  aa95 a882 b766 829c 0020 b0de 0000 0000  .....f..........
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000                                ....
21:06:51.204191 IP 192.168.1.1 > 192.168.1.12: ICMP time exceeded in-transit,
        length 60
        0x0000:  45c0 0050 a202 0000 4001 548d c0a8 0101  E..P....@.T.....
        0x0010:  c0a8 010c 0b00 09fe 0000 0000 4500 0034  ............E..4
        0x0020:  b768 0000 0111 ed84 c0a8 010c aa95 a882  .h..............
        0x0030:  b766 829c 0020 b0de 0000 0000 0000 0000  .f..............
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................

Note that traceroute sends out UDP messages, starting at port 33435 and incrementing the port number by one with each additional message. The port number is incremented so that it’s possible later to reconstruct the order in which the packets are sent. Note that the ICMP packet from offset 0x001C onward contains the original UDP packet. As noted earlier, ICMP messages need to use a number of different techniques to provide context—error messages such as “TTL exceeded” include the IP header and the first 8 bytes of the original packet. This includes the UDP source port number. traceroute uses the included UDP source port to reconstruct the order in which these ICMP messages were sent.

While traceroute uses UDP by default, the same technique can be used by TCP or any other protocol where there is a controllable value (such as an ephemeral port number) in the first 8 bytes of the IP payload.

ping and traceroute are more useful if you can use them from different locations. To that end, a number of internet service providers and other organizations provide looking-glass servers. A looking-glass server is a publicly accessible (generally via the web) interface to any of a number of common internet applications. Most looking glasses are managed by NOCs or ISPs, and provide access to multiple routers. There is no standard for implementation, and different looking glasses will provide different services. A comprehensive list is available at www.traceroute.org.

Using nc as a Swiss Army Multitool

ping and traceroute represent traffic on the network proper; if you want to communicate with a specific service, you have to have a service client available to you. Alternatively, you can use netcat. In Chapter 5, I touched briefly on this tool: let’s talk about what it can do in more depth.

Basic netcat invocation is simply netcat (or nc) with the destination address and port. If this connects to an open port, then netcat will operate like the cat command. If you know what the targeted service operates like, you can then send a request and get a response. For example, you can run an HTTP request by hand:

$ echo "GET /" | nc www.google.com 80 | head -4
HTTP/1.0 200 OK
Date: Mon, 20 Feb 2017 16:51:48 GMT
Expires: -1
Cache-Control: private, max-age=0

You can quickly implement a fast horizontal scan by using netcat’s zero I/O mode (-z). Zero I/O simply tries to open a port and, if it succeeds, immediately closes the session. On its own, this results in netcat providing zero information, so you need to up the output using netcat’s verbose option (-v):

$ netcat -vz localhost 8000
localhost [127.0.0.1] 8000 (irdmi) open
$ netcat -vz localhost 122
localhost [127.0.0.1] 122 (smakynet): Connection refused

In the absence of other tools, netcat can be used to scan horizontally or vertically. The short and sour way to do this is just to iterate at the command line, possibly using the -w option to slow down consecutive invocations. For example, if I want to scan a local /24, I can do:

i=0; while [ ${i} -le 256 ];
do
   netcat -vz -w 2 192.168.1.${i} 8000
   i=$[ ${i} + 1 ]
done

Which results in:

192.168.1.1 8000 (irdmi): Connection refused
192.168.1.2 8000 (irdmi) open
192.168.1.3 8000 (irdmi): Operation timed out

and so on. Note the refusals and timeouts—a refusal means a host was present and the port was closed, while a timeout means that no successful communication took place (which is usually a good indicator that there is nothing on the other side). You can swap in a vertical scan by changing the IP address to the port number. netcat will take ranges:

$ netcat -vz 192.168.1.13 7990-8080
192.168.1.13 7990-8080 192.168.1.13 8000 (irdmi) open

Note that netcat will only provide output for open ports.

nmap Scanning for Discovery

nc is versatile, but not specialized. nmap integrates both discovery and assessment roles into an open source package.

Let’s start our nmap exercises by doing some simple horizontal scanning for discovery (we’ll get to assessment in the next section). These nmap commands are equivalent to the scans we did by hand with nc. Let’s start with a horizontal scan on a /24; this is done using -p to specify a port, and then an address range. nmap can take CIDR notation (as seen here) as well as using a dash, so I could express this as 192.168.1.0-255:¹

# Scanning a /24 with nmap; some addresses removed for brevity
$ nmap -p 8000 192.168.1.0/24

Starting Nmap 7.40 ( https://nmap.org ) at 2017-02-13 12:08 EST
Nmap scan report for 192.168.1.1
Host is up (0.0024s latency).
PORT     STATE  SERVICE
8000/tcp closed http-alt

Nmap scan report for 192.168.1.2
Host is up (0.015s latency).
PORT     STATE  SERVICE
8000/tcp closed http-alt

Nmap scan report for 192.168.1.13
Host is up (0.00021s latency).
PORT     STATE SERVICE
8000/tcp open  http-alt

Nmap done: 256 IP addresses (8 hosts up) scanned in 2.89 seconds

By default, nmap scans privileged TCP ports (1–1024), and reports anything open. For example:

$ nmap 192.168.1.13

Starting Nmap 7.40 ( https://nmap.org ) at 2017-02-13 12:17 EST
Nmap scan report for 192.168.1.13
Host is up (0.00046s latency).
Not shown: 813 closed ports, 182 filtered ports
PORT     STATE SERVICE
53/tcp   open  domain
445/tcp  open  microsoft-ds
548/tcp  open  afp
631/tcp  open  ipp
8000/tcp open  http-alt

You can specify ports using the -p option, as shown here:

$ nmap -p 53,445,546-550  192.168.1.13

Starting Nmap 7.40 ( https://nmap.org ) at 2017-02-13 12:25 EST
Nmap scan report for 192.168.1.13
Host is up (0.00050s latency).
PORT    STATE  SERVICE
53/tcp  open   domain
445/tcp open   microsoft-ds
546/tcp closed dhcpv6-client
547/tcp closed dhcpv6-server
548/tcp open   afp
549/tcp closed idfp
550/tcp closed new-rwho

Note that when you specify the ports, the output will include only open ports. The default scan and nmap’s fast scan (-F, it only scans the 100 most commonly used ports) both only show open ports.

That’s the basics of horizontal and vertical TCP scanning with nmap. At this point, simple discovery scanning mostly involves knowing a couple of other options: in particular, -sU (scan UDP), -n (drop DNS resolution), and -sn (simple ping, akin to the netcat zero I/O mode).

Assessment: nmap, a Bunch of Clients, and a Lot of Repositories

Following discovery is assessment: figuring out what a host is vulnerable to. Effectively assessing a host’s vulnerabilities requires inventory data and expertise—inventory to know what services and versions are running on a host, and expertise to know how someone can exploit this.

In this section, I will discuss a number of inventory techniques to support assessment. We will begin with using nmap to identify version numbers and operating systems, then discuss the general process of figuring out what a host is running. Finally, we will discuss the problem of determining what services are running on an open port when you have no idea what’s there.

Basic Assessment with nmap

The basic command-line arguments for assessment are -O (check the operating system) and -sV (check the service and version on open ports). These options are frontends to what is, effectively, an expert system: a decision tree that passes different combinations of packets and flags in order to determine how your system responds and what that implies.

Running -O will provide you with an OS profile. If nmap knows the OS (and it knows a lot), you’ll get a simple output like this one:

$ nmap -O
Nmap scan report for 192.168.1.13
Host is up (0.000070s latency).
Not shown: 995 closed ports
PORT     STATE SERVICE
53/tcp   open  domain
445/tcp  open  microsoft-ds
548/tcp  open  afp
631/tcp  open  ipp
8000/tcp open  http-alt
Device type: general purpose
Running: Apple OS X 10.10.X|10.11.X
OS CPE: cpe:/o:apple:mac_os_x:10.10 cpe:/o:apple:mac_os_x:10.11
OS details: Apple OS X 10.10 (Yosemite) - 10.11 (El Capitan) (Darwin 14.0.0 -
    15.4.0)

Most of the report was discussed earlier—the new fields are “Device type,” “Running,” and the OS information. The device type refers to what type of device nmap thinks it is; “general purpose” indicates that it’s a general-purpose computing machine (i.e., a laptop or desktop), while other types include terminals, networking hardware (routers and switches), and various types of enterprise hardware.² “Running” simply refers to nmap’s guess as to what type of operating system is running; nmap makes this guess with an ordered table of probabilities.

The next line of note is the CPE (Common Platform Enumeration). The OS information includes the Common Platform Enumeration (CPE), discussed in “The NVD and MITRE Standards”, and details on the operating system in use.

An Example: Finding an Embedded System

When nmap doesn’t recognize what a particular system is, it sends back a lot of information. For example, suppose I run an nmap scan and get this back:

Nmap scan report for 192.168.1.6
Host is up (0.0011s latency).
Not shown: 999 closed ports
PORT   STATE SERVICE
80/tcp open  http
MAC Address: 00:17:88:11:22:47 (Philips Lighting BV)
No exact OS matches for host (If you know what OS is running on it, see
https://nmap.org/submit/ ).
TCP/IP fingerprint:
OS:SCAN(V=7.40%E=4%D=2/21%OT=80%CT=1%CU=35895%PV=Y%DS=1%DC=D%G=Y%M=001788%T
OS:M=58ACE89C%P=x86_64-apple-darwin16.3.0)SEQ(SP=A2%GCD=1%ISR=DC%TI=I%CI=I%
OS:II=RI%SS=S%TS=U)SEQ(SP=AB%GCD=1%ISR=DC%TI=I%II=RI%SS=O%TS=U)OPS(O1=M218%
OS:O2=M218%O3=M218%O4=M218%O5=M218%O6=M218)WIN(W1=860%W2=860%W3=860%W4=860%
OS:W5=860%W6=860)ECN(R=Y%DF=N%T=FF%W=860%O=M218%CC=N%Q=)T1(R=Y%DF=N%T=FF%S=
OS:O%A=S+%F=AS%RD=0%Q=)T2(R=N)T3(R=N)T4(R=Y%DF=N%T=FF%W=860%S=A%A=S%F=AR%O=
OS:%RD=0%Q=)T4(R=N)T5(R=Y%DF=N%T=FF%W=860%S=A%A=S+%F=AR%O=%RD=0%Q=)T6(R=Y%D
OS:F=N%T=FF%W=860%S=A%A=S%F=AR%O=%RD=0%Q=)T6(R=N)T7(R=N)U1(R=Y%DF=N%T=FF%IP
OS:L=38%UN=0%RIPL=G%RID=G%RIPCK=G%RUCK=G%RUD=G)IE(R=Y%DFI=N%T=FF%CD=S)

Network Distance: 1 hop

OS detection performed. Please report any incorrect results at
https://nmap.org/submit/ .

This is actually a fingerprint that nmap creates to feed back into its knowledge base. This is actually a packed, fixed-length format.³ While it’s great for nmap (and you should submit it if you’re comfortable sharing—this profiling will improve only if you share!), there’s still the question of what it means for us.

First, note the MAC address. The MAC address is registered to a Philips lighting device; a network interface that isn’t from a company specializing in network interfaces is likely to be some kind of IoT thing.

Next, note that the scan also includes a list of open ports, and tcp/80 is open for business. Given that, I go and run nc on port 80:

$ echo "GET /" | nc 192.168.1.6 80 HTTP/1.1 200 OK
Content-type: text/html

<html><head><title>hue personal wireless lighting</title></head><body>
<b>Use a modern browser to view this resource.</b></body></html>

And yeah, it’s a hub for a Philips Hue wireless lighting system. As a rule of thumb, running a quick GET / on an open port can’t hurt—it’s obvious on port 80, but odds are good you’ll get a meaningful response on other ports as well. Most servers these days are modified web servers of some variety, so you’re likely to find something speaking HTTP on the other end.

The NVD and MITRE Standards

For years, the US government contracted the MITRE Corporation to maintain a number of different repositories on attacks, platforms, vulnerabilities, and other information on security. The most well-known of these is the Common Vulnerability Enumeration (CVE), a living repository of software vulnerabilities. MITRE has produced a number of other standards as well, with the CPE mentioned earlier following just behind the CVE in exposure. The value of all of these standards is that they provide a common point of reference. In a field where people can get possessive about nomenclature, this is not to be underestimated.

As a rule, the most common and important standards are now managed by NIST under the header of the National Vulnerability Database (NVD). Here’s a closer look at these standards:

The Common Vulnerability Enumeration (CVE) is an index of software vulnerabilities. CVE indices are of the form CVE-YYYY-NNNN, where YYYY is the year of assignment and NNNN is a numeric index. CVEs are maintained as part of the NVD, and are accessible at https://web.nvd.nist.gov/view/vuln/search.
The Common Platform Enumeration (CPE) is a hierarchical name structure that begins with the identifier “cpe”, followed by one or more values separated by colons; these values are, in order, the part, the vendor, the product, the version, the update, the edition, the language, the software edition, the target software, the target hardware, and anything else. Some fields are optional, as seen in the cpe:/o:apple:mac_os_x:10.10 string in the nmap -O output shown earlier (part: operating system, vendor: Apple, product: Mac OS X, version: 10.10; everything else dropped). CPEs are also part of the NVD; the standard and dictionaries are at https://nvd.nist.gov/cpe.cfm.
The Security Content Automation Protocol (SCAP) is an umbrella term encompassing the CVE and CPE as well as the Common Weakness Enumeration (CWE), the Common Vulnerability Scoring System (CVSS), and a number of other standards.

Vulnerability assessment involves taking a very specific inventory (down to platform and version numbers), consulting intelligence sources such as the NVD to determine which vulnerabilities exist, and then determining the response—patching, blocking, taking systems down, or whatever other action is deemed appropriate.

Unpatchable Vulnerabilities

An important note here: while we usually talk about vulnerabilities as flaws in software design, a flaw can also appear in policy or be embedded in the design of the protocol, and it may not be fixable. Some examples:

Mediocre password management, in particular the use of default passwords for administration accounts, is a universal headache. The recent Mirai botnet exploited this across multiple embedded systems.
Reflection attacks are a common problem with UDP-based protocols such as NTP and DNS. Reflection attacks leverage UDP’s statelessness and the willingness of these protocols to send back “I dunno” packets when receiving uncertain messages.
Administrative tools are often embedded into default installations with minimal, if any, permissions to execute them. The entire field of malware free intrusions depends on this. Finally, when dealing with embedded and IoT systems, expect that you often won’t be able to fix the vulnerability, because patching and maintenance isn’t part of the design.

The most advanced assessment tool that nmap provides is a full Lua-based flexible scripting capability, the Nmap Scripting Engine (NSE). The NSE commands nmap’s capabilities using Lua scripts that can write complex, sequential analyses of particular hosts and enable authors to identify vulnerabilities and traffic features that are outside the capabilities of vanilla nmap. The NMAP project maintains a constantly updated list of NSE scripts; it’s worth checking on a regular basis for specific and current vulnerabilities.

What’s important to recognize about all of these tools is that the mechanical components are less important than the expertise. It’s critical to pay attention to what features and scripts appear that are relevant to your particular organization. Similarly, if you buy a vulnerability scanner, you’re paying for the company to keep up-to-date on current vulnerabilities. In Chapter 17 we discuss these issues in more detail.

Heartbeat Signals

The first thing to do in any measurement is check your ruler⁴; when you see an asset disappear on a network, your first step, before all the probing and pinging, is to ensure that the asset is the problem and not your data collection system. With critical assets, it’s better to make this process proactive, by including some kind of beacon. A heartbeat signal is a process that periodically sends out a packet simply to remind you that the host is alive and kicking.

Heartbeat signals do not have to be particularly elaborate, but there are a couple of notes on setting them up properly that can help. When setting up heartbeat signals, pay attention to inventory and instrumentation, minimize secondary network effects, and pick a useful time interval.

Regarding inventory: the goal of the heartbeat signal is to provide an analyst with a quick heads up within a data feed. Since you can craft the packet to your needs within the constraints of the feed, feel free to leverage that—embed UUIDs in the packet, send packets to nonexistent hosts (the goal is for your detection systems to recognize it, not necessarily any target in the network).

That said, limit the secondary effects if possible. For defenders who are unaware what the signals are, heartbeat signals are going to be a deliciously obvious anomaly, a problem exacerbated if they generate a lot of noise due to TCP or ICMP responses. I prefer sending UDP packets because they limit the amount of noise that TCP or ICMP echo requests will send out.

Using Active Vantage Data for Verification

Because analysts have limited resources, the most effective use for active domain data is as a concentrated, limited-scope and high-information complement to broader passive data collection. Probe actively when the passive system can’t figure out what’s going on.

To make this more concrete, consider a scenario where you passively collect inventory to guide incident response. Using the techniques discussed in Chapter 19, you can passively collect information on hosts within the network. However, when doing so you will find that there are sites you cannot passively examine—examples include HTTPS servers (encrypted content) and services on unidentified ports. In this case, you can set up an automated watch for the presence of these services and, when identified, automatically scan them to find out more about what they are doing.

Passive and active discovery complement each other. Passive discovery is very handy when you don’t know much about your network, and will identify hosts that you don’t know about. Active discovery will help you find out what hosts are actually running.

Another example of how to combine active and passive mapping uses dropouts to identify diagnostic concerns. In this case, you use passive detection to monitor hosts within the network. For example, you may monitor the outgoing traffic from your servers and establish a lower limit for volume before you’re concerned that that server is not communicating (e.g., if there’s less than 5 MB every 10 minutes, or the heartbeat signal is not present). When you encounter a disruption in those timeouts, you initiate an active check.

In both cases, you rely on a large flow-based collection system to trigger a specific, fine-grained probe. This is, in my experience, the best way to combine active and passive probing—use the passive techniques to determine the network’s population, and where you encounter ambiguity or uncertainty, probe actively.

Table of Contents for
Network Security Through Data Analysis, 2nd Edition

Chapter 7. Data and Sensors in the Active Domain

Discovery, Assessment, and Maintenance

Discovery: ping, traceroute, netcat, and Half of nmap

Checking Connectivity: Using ping to Connect to an Address

Example 7-1. ping output

Tracerouting

Using nc as a Swiss Army Multitool

nmap Scanning for Discovery

Assessment: nmap, a Bunch of Clients, and a Lot of Repositories

Basic Assessment with nmap

Using Active Vantage Data for Verification

Further Reading

Table of Contents for Network Security Through Data Analysis, 2nd Edition

Chapter 7. Data and Sensors in the Active Domain

Discovery, Assessment, and Maintenance

Discovery: ping, traceroute, netcat, and Half of nmap

Checking Connectivity: Using ping to Connect to an Address

Example 7-1. ping output

Tracerouting

Using nc as a Swiss Army Multitool

nmap Scanning for Discovery

Assessment: nmap, a Bunch of Clients, and a Lot of Repositories

Basic Assessment with nmap

Using Active Vantage Data for Verification

Further Reading

Table of Contents for
Network Security Through Data Analysis, 2nd Edition