Commands and Arguments

The basic operation of bash is to execute a command, that is, to run another program. When several words appear on the command line bash assumes that the first word is the name of the program to run and the remaining words are the arguments to the command. For example:

mkdir -p /tmp/scratch/garble

will have bash run the command called mkdir and it will pass it two arguments -p and /tmp/scratch/garble. By convention programs generally put their options first, and have them begin with a leading "-“, as is the case here with the -p option. This particular command will create a directory called /tmp/scratch/garble. The -p option will mean that no errors will be reported and any intervening directories will be created (or attempted) as needed (e.g., if only /tmp exists, it will create /tmp/scratch before attempting to create /tmp/scratch/garble).

Standard Input/Output/Error

A running program is called a process and every process in the Unix/Linux/Posix (and thus Windows) environment has three distinct input/output file descriptors. These three are called “standard input” (or stdin, for short), “standard output” (stdout), and “standard error” (stderr).

As you might guess by its name, stdin is the default source for input to a program, by default the characters coming from the keyboard. When your script reads from stdin it is reading characters typed on the keyboard or (as we shall see shortly) it can be changed to read from a file. Stdout is the default place for sending output from a program. By default the output appears in the window which is running your shell or shell script. Standard error can also be sent output from a program, but it is (or should be) where error messages are written. It’s up to the person writing the program to direct any output to either stdout or stderr. So be conscientious when writing your scripts to send any error messages not to stdout but to stderr (as shown below).

Redirection and Piping

One of the great innovations of the shell was that it gave you a mechanism whereby you could take a running program and change where it got its input and/or change where it sent its output without modifying the program itself. If you have a program called handywork and it reads its input from stdin and writes its results to stdout, then you can change its behavior as simply as this:

handywork < data.in  > results.out

which will run handywork but will have the input come not from the keyboard but instead from the data file called data.in (assuming such a file exists and has input in the format we want). Similarly the output is being sent not to the screen but into a file called results.out (which will be created if it doesn’t exist and overwritten if it does). This technique is called “redirection” because we are re-directing input to come from a different place and re-directing output to go somewhere other than the screen.

What about stderr? The syntax is similar. We have to distinguish between stout and stderr when redirecting data coming out of the program and we make this distinction through the use of the file descriptor numbers. Stdin is file descriptor 0, stdout is file descriptor 1 and stderr is file descriptor 2 so we can redirect error messages this way:

handywork 2> err.msgs

which will redirect only stderr and send any such error message output to a file we called err.msgs (for obvious reasons).

Of course we can do all three on the same line:

handywork < data.in  > results.out  2> err.msgs

Sometimes we want the error messages combined with the normal output (as it does by default when both are written to the screen). We can do this with the following syntax:

handywork < data.in  > results.out 2>&1

which says to send stderr (2) to the same location as file descriptor 1 (”&1“). Note that without the ampersand, the error messages would just be sent to a file named “1”. This combining of stdout and stderr is so common that there is a useful shorthand notation:

handywork < data.in  &> results.out

If you want to discard standard output you can redirect it to a special file called /dev/null as follows:

handywork < data.in > /dev/null

To view output on the command line and simultaneously redirect that same output to a file, use the tee command. The following will display the output of handywork to the screen and also save it to results.out:

handywork < data.in | tee results.out

A file will be created or truncated (i.e., contents discarded) when output is redirected. If you want to preserve the file’s existing content you can, instead, append to the file using a double greater-than sign, like this:

handywork < data.in  >> results.out

This will execute handywork and then any output from stdout will be appended to the file results.out rather than overwriting its existing content.

Similarly this command line:

handywork < data.in  &>> results.out

will execute handywork and then append both stdout and stderr to the file results.out rather than overwriting its existing content.

Running Commands in the Background

Throughout this book we will be going beyond one-line commands and will be building complex scripts. Some of these scripts can take a significant amount of time to execute, so much so that you may not want to spend time waiting for them to complete. Instead, you can run any command or script in the background using the & operator. The script will continue to run, but you can issue other commands and/or run other scripts. For example, to run ping in the background and redirect standard output to a file:

ping 192.168.10.56 > ping.log &

You will likely want to redirect standard output and/or standard error to a file when sending tasks to the background, or the task will continue to print to the screen and interrupt other activities you are performing.

You can use the jobs command to list any tasks currently running in the background.

$ jobs
[1]+  Running                 ping 192.168.10.56 > ping.log &

Use the fg command and the corresponding job number to bring the task back into the foreground.

$ fg 1
ping 192.168.10.56 > ping.log

If your task is currently executing in the foreground you can use CTRL-Z to suspend the process and then bg to continue the process in the background. From there you can use jobs and fg as described above.

From Command Line to Script

A shell script is just a file that contains the same commands that you could type on the command line. Put one or more commands into a file and you have a shell script. If you called your file myscript you can run that script by typing: bash myscript or you can give it “execute permission” (e.g., chmod 755 myscript) and then you can invoke it directly: ./myscript to run the script. We often include the following line as the first line of the script, which tells the operating system which scripting language we are using:

#!/bin/bash -

Of course this assumes that bash is located in the /bin directory. If your script needs to be more portable, you could use this approach instead:

#!/usr/bin/env bash

It uses the env command to look up the location of bash and is considered the standard way to address the portability problem. It makes the assumption, however, that the env command is to be found in /usr/bin.

Summary

In this chapter you saw how to run single commands and redirect input and output. In the next chapter we will discuss the real power of scripting, which comes from being able to run commands repeatedly, make decisions in the script, and loop over a variety of inputs.

Exercises

1. Write a command that executes ifconfig and redirects standard output to a file named ipaddress.txt.

2. Write a command that executes ifconfig and redirects standard output and appends it to a file named ipaddress.txt.

3. Write a command that copies all of the files in the directory /etc/a to the directory /etc/b and redirects standard error to the file copyerror.log.

4. Write a command that performs a directory listing (ls) on the root file directory and pipes the output into the more command.

5. Write a command that executes mytask.sh and sends it to the background.

6. Given the job list below, write the command that brings the Amazon ping task to the foreground.

   [1]   Running                 ping www.google.com > /dev/null &
   [2]-  Running                 ping www.amazon.com > /dev/null &
   [3]+  Running                 ping www.oreilly.com > /dev/null &

Output

As with any programming language, bash has the ability to output information to the screen. Output can be achieved using the echo command.

$ echo "Hello World"

Hello World

You may also use the printf command which allows for some additonal formatting.

$ printf "Hello World"

Hello World

Variables

Bash variables begin with an alphabetic character or underscore followed by alphanumeric characters. They are string variables unless declared otherwise. To assign a value to the variable, you write something like this:

MYVAR=textforavalue

To retrieve the value of that variable, for example to print out the value using the echo command, you use the $ in front of the variable name, like this:

echo $MYVAR

If you want to assign a series of words to the variable, that is, to preserve any whitespace, use quotation marks around the value, as in:

MYVAR='here is a longer set of words'
OTHRV="either double or single quotes will work"

The use of double quotes will allow other substitutions to occur inside the string. For example:

firstvar=beginning
secondvr="this is just the $firstvar"
echo $secondvr

This will result in the output: this is just the beginning

There are a variety of substitutions that can occur when retrieving the value of a variable; we will show those as we use them in the scripts to follow.

You can also store the output of a shell command using $( ) as follows:

CMDOUT=$(pwd)

That will execute the command pwd in a sub-shell and rather than printing the the result to stdout, it will store the output of the command in the variable CMDOUT. You can also pipe together multiple commands within the $ ( ).

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# echoparams.sh
#
# Description:
# Demonstrates accessing parameters in bash
#
# Usage:
# ./echoparms.sh <param 1> <param 2> <param 3>
#

echo $#
echo $0
echo $1
echo $2
echo $3

$ ./echoparams.sh bash is fun

3
./echoparams.sh
bash
is
fun

Input

User input is received in bash using the read command. The read command will obtain user input from the command line and store it in a specified variable. The script below reads user input into the MYVAR variable and then prints it to the screen.

read MYVAR
echo "$MYVAR"

Conditionals

Bash has a rich variety of conditionals. Many, but not all, begin with the keyword if.

Any command or program that you invoke in bash may do some output but it will also always return a success or fail value. In the shell this value can be found in the $? variable immediately after a command has run. A return value of 0 is considered “success” or “true”; any non-zero value is considered “error” or “false”. The simplest form of the if statement uses this fact. It takes the form:

if cmd
then
   other cmds
fi

For example, the script below attempts to change directories to /tmp. If that command is successful (returns 0) the body of the if statement will execute.

if  cd /tmp
then
    echo "here is what is in /tmp:"
    ls -l
fi

Bash can even handle a pipeline of commands in a similar fashion:

if ls | grep pdf
then
    echo found one or more pdf files here
fi

With a pipeline, it is the success/failure of the last command in the pipeline that determines if the “true” branch is taken. Here is an example where that fact matters:

ls | grep pdf | wc

This series of commands will be “true” even if no pdf string is found by the grep command. That is because the wc command (a word count of the input) will print:

0       0       0

That output indicates 0 characters, 0 words, and 0 lines when no output comes from the grep command. That is still a successful (or true) result, not an error or failure. It counted as many lines as it was given, even if it was given zero lines to count.

A more typical form of if used for comparisons makes use of the compound command [[ or the shell built-in commands [ or test. Use these to test file attributes or to make comparisons of value.

To test if a file exists on the file system:

if [[ -e $FILENAME ]]
then
    echo $FILENAME exists
fi

Table 2-1 lists additional tests that can be done on files using if comparisons.

To test if the variable $VAL is less than the variable $MIN:

if [[ $VAL -lt $MIN ]]
then
    echo "value is too small"
fi

Table 2-2 lists additional numeric tests that can be done using if comparisons.

if [[ $VAL < $OTHR ]]

If you want to do numerical comparisons with the less-than sign, use the double parentheses construct. It assumes that the variables are all numerical and will evaluate them as such. Empty or unset variables are evaluated as 0. Inside the parentheses you don’t need the $ operator to retrieve a value, except for positional parameters like $1 and $2 (so as not to confuse them with the constants 1 and 2). For example:

if (( VAL < 12 ))
then
    echo "value $VAL is too small"
fi

In bash you can even make branching decisions without an explicit if/then construct. Commands are typically separated by a newline - that is, they appear one per line. You can get the same effect by separating them with a semicolon. If you write cd $DIR ; ls then bash will perform the cd and then the ls.

Two commands can also be separated by either && or || symbols. If you write cd $DIR && ls then the ls command will run only if the cd command succeeds. Similarly if you write cd $DIR || echo cd failed the message will be printed only if the cd fails.

You can use the [[ syntax to make various tests, even without an explicit if.

[[ -d $DIR ]] && ls "$DIR"

means the same as if you had written

if [[ -d $DIR ]]
then
  ls "$DIR"
fi

[[ -d $DIR ]] || echo "error: no such directory: $DIR" ; exit

[[ -d $DIR ]] || { echo "error: no such directory: $DIR" ; exit ; }

Looping

Looping with a while statement is similar to the if construct in that it can take a single command or a pipeline of commands for the decision of true or false. It can also make use of the brackets or parentheses as in the if examples, above.

In some languages braces ( { } ) are used to group the statement together that are the body of the while loop. In others, like python, indentation is the indication of which statements are the loop body. In bash, however, the statements are grouped between two keywords: do and done.

Here is a simple while loop:

i=0
while (( i < 1000 ))
do
    echo $i
    let i++
done

The loop above will execute while the variable i is less than 1000. Each time the body of the loop executes it will print the value of i to the screen. It then uses the let command to execute i++ as an arithmetic expression, thus incrementing i by 1 each time.

Here is a more complicated while loop that executes commands as part of its condition.

while ls | grep -q pdf
do
    echo -n 'there is a file with pdf in its name here: '
    pwd
    cd ..
done

A for loop is also available in bash - in three variations.

Simple numerical looping can be done using the double parentheses construct. It looks much like the for loop in C or Java, but with double parentheses and with do and done instead of braces:

for ((i=0; i < 100; i++))
do
    echo $i
done

Another useful form of the for loop is used to iterate through all the parameters that are passed to a shell script (or function within the script), that is, $1, $2, $3 an so on. Note that ARG in args.sh can be replaced with any variable name of your choice.

for ARG
do
    echo here is an argument: $ARG
done

Here is the output of args.sh when three parameters are passed in.

$ ./args.sh bash is fun

here is an argument: bash
here is an argument: is
here is an argument: fun

Finally, for an arbitrary list of values, use a similar form of the for statement simply naming each of the values you want for each iteration of the loop. That list can be explicitly written out, like this:

for VAL in 20 3 dog peach 7 vanilla
do
    echo $VAL
done

The values used in the for loop can also be generated by calling other programs or using other shell features:

for VAL in $(ls | grep pdf) {0..5}
do
    echo $VAL
done

Here the variable VAL will take, in turn, the value for each of the filenames that ls piped into grep finds with the letters pdf in its filename (e.g. “doc.pdf” or “notapdfile.txt”) and then each of the numbers 0 through 5. It may not be that sensible to have the variable VAL be a filename sometimes and a single digit another time, but this shows you that it can be done.

Functions

Define a function with sytnax like this:

function myfun ()
{
  # body of the function goes here
}

Not all that syntax is necessary - you can use either "function" or "()" - you don’t need both. We recommend, and will be using, both - mostly for readability.

There are a few important considerations to keep in mind with bash functions:

• Unless declared with the local builtin command inside the function, variables are global in scope. A for loop which sets and increments i could be messing with the value of i used elsewhere in your code.

• The braces are the most commonly used grouping for the function body, but any of the shell’s compound command syntax is allowed - though why, e.g., would you want the function to run in a sub-shell?

• Redirecting I/O on the braces does so for all the statements inside the function. Examples of this will be seen in upcomoing chapters.

• No parameters are declared in the function definition. Whatever and however many arguments are supplied on the invocation of the function are passed to it.

The function is called (invoked) just like any command is called in the shell. Having defined myfun as a function you can call it like this:

myfun 2 /arb "14 years"

which calls the function myfun supplying it with 3 arguments.

Pattern Matching in bash

When you need to name a lot of files on a command line, you don’t need to type each and every name. Bash provides pattern matching (sometimes called “wildcarding”) to allow you to specify a set of files with a pattern. The easiest one is simply an asterisk * (or “star”) which will match any number of any characters. When used by itself, therefore, it matches all files in the current directory. The asterisk can be used in conjunction with other characters. For example \*.txt matches all the files in the current directory which end with the four characters .txt. The pattern /usr/bin/g\* will match all the files in /usr/bin that begin with the letter g.

Another special character in pattern matching is ? the question mark, which matches a single character. For example, source.? will match source.c or source.o but not source.py or source.cpp.

The last of the three special pattern matching characters are [ ], the square brackets. A match can be made with any one of the characters listed inside the square brackets, so the pattern x[abc]y matches any or all of the files named xay, xby, or xcy, assuming they exist. You can specify a range within the square brackets, like [0-9] for all digits. If the first character within the brackets is either a \! or a ^ then the pattern means anything other than the remaining characters in the brackets. For example, [aeiou] would match a vowel whereas [^aeiou] would match any character except the vowels (including digits and punctuation characters).

Similar to ranges, you can specify character classes within braces. Table 2-3 lists the character classes and their description.

Character classes are specified like this: [:cntrl:] within square brackets (so you have two sets of []). For example, this pattern: \*[[:punct:]]jpg will match any filename that has any number of any characters followed by a punctuation character followed by the letters jpg. So it would match files named wow!jpg or some,jpg or photo.jpg but not a file named this.is.myjpg since there is no punctuation character right before the jpg.

There are more complex aspects of pattern matching if you turn on the shell option extglob (like this: shopt -s extglob) so that you can repeat patterns or negate patterns. We won’t need these in our example scripts but we encourage you to learn about them (e.g., via the bash man page).

There are a few things to keep in mind when using shell pattern matching:

• Patterns aren’t regular expressions (discussed later); don’t confuse the two.

• Patterns are matched against files in the file system; if the pattern begins with a pathname (e.g., /usr/lib ) then the matching will be done against files in that directory.

• If no pattern is matched, the shell will use the special pattern matching characters as literal characters of the filename; for example, if your script says echo data > /tmp/*.out but there is no file in /tmp that ends in .out then the shell will create a file called *.out in the /tmp directory. Remove it like this: rm /tmp/\*.out by using the backslash to tell the shell not to pattern match with the asterisk.

• No pattern matching occurs inside of quotes (either double or single quotes), so if your script says echo data > "/tmp/*.out" it will create a file called /tmp/*.out (which we recommend you avoid doing).

Writing Your First Script - Detecting Operating System Type

Now that we have gone over the fundamentals of the command line and bash you are ready to write your first script. The bash shell is available on a variety of platforms including Linux, Windows, macOS, and Git Bash. As you write more complex scripts in the future it is imperative that you know what operating system you are interacting with as each one has a slightly different set of commands available. The osdetect.sh script helps you in making that determination.

The general idea of the script is that it will look for a command that is unique to a particular operating system. The limitation is that on any given system an administrator may have created and added a command with that name, so this is not foolproof.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# osdetect.sh
#
# Description:
# Distinguish between MS-Windows/Linux/MacOS
#
# Usage:
# Output will be one of: Linux MSWin macOS
#

if type -t wevtutil &> /dev/null           
then
    OS=MSWin
elif type -t scutil &> /dev/null           
then
    OS=macOS
else
    OS=Linux
fi
echo $OS

We use the type built-in in bash to tell us what kind of a command (alias, keyword, function, built-in, or file) its arguments are. The -t option tells it to print nothing if the command isn’t found. The command returns as “false” in that case. We redirect all the output (both stdout and stderr) to /dev/null thereby throwing it away, as we only want to know if the wevtutil command was found.

Again we use the type built-in but this time we are looking for the scutil command which is available on macOS systems.

Summary

The bash shell can be seen as a programming language, one with variables and if/then/else statements, loops, and functions. It has its own syntax, similar in many ways to other programming languages, but just different enough to catch you if you’re not careful.

It has its strengths - like easily invoking other programs or connecting sequences of other programs - and it has its weaknesses: it doesn’t have floating point arithmetic or much support (though some) for complex data structures.

In the chapters ahead we will describe and use many bash features and OS commands in the context of cybersecurity operations. We will further explore some of the features we have touched on here, and other more advanced or obsure features. Keep your eyes out for those featues and practice and use them for your own scripting.

Exercises

1. Experiment with the uname command, seeing what it prints on the various operating systems. Re-write the osdetect.sh script to use the uname command, possibly with one of its options. Caution: not all options are available on every operating system.

2. Modify the osdetect.sh script to use a function. Put the if/then/else logic inside the function and then call it from the script. Don’t have the function itself do any output. Make the output come from the main part of the script.

3. Set the permissions on the osdetect.sh script to be executable (see man chmod) so that you can run the script without using bash as the first word on the command line. How do you now invoke the script?

4. Write a script called argcnt.sh that tells how many arguments are supplied to the script.

   1. Modify your script to have it also echo each argument one per line.

   2. Modify your script further to label each argument like this:

      $ bash argcnt.sh this is a "real live" test
      there are 5 arguments
      arg1: this
      arg2: is
      arg3: a
      arg4: real live
      arg5: test
      $

5. Modify argcnt.sh so it only lists the even arguments.

Commands in Use

We introduce the grep family of commands to demonstrate the basic regex patterns.

grep -R -i 'password' /home

Regular Expression Metacharacters

Regular expressions are patterns that are created using a series of characters and metacharacters. Metacharacters such as "?" and "*" have special meaning beyond their literal meaning in regex.

$ grep 'T.o' frost.txt

1    Two roads diverged in a yellow wood,

$ egrep 'T.?o' frost.txt

1    Two roads diverged in a yellow wood,
5    To where it bent in the undergrowth;

$ grep 'T.*o' frost.txt

1    Two roads diverged in a yellow wood,
5    To where it bent in the undergrowth;
7 Excerpt from The Road Not Taken by Robert Frost

$ egrep 'T.+o' frost.txt

1    Two roads diverged in a yellow wood,
5    To where it bent in the undergrowth;
7 Excerpt from The Road Not Taken by Robert Frost

$ egrep 'And be one (stranger|traveler), long I stood' frost.txt

3    And be one traveler, long I stood

$ grep -P '\d' frost.txt

1    Two roads diverged in a yellow wood,
2    And sorry I could not travel both
3    And be one traveler, long I stood
4    And looked down one as far as I could
5    To where it bent in the undergrowth;
6
7 Excerpt from The Road Not Taken by Robert Frost

grep 'X[[:upper:][:digit:]]' idlist.txt

User: XTjohnson
an XWing model 7
an X7wing model

1    Command
2    <i>line</i>
3    is
4    <div>great</div>
5    <u>!</u>

$ egrep '<([A-Za-z]*)>.*</\1>' tags.txt

2    <i>line</i>
4    <div>great</div>
5    <u>!</u>

Summary

Regular expressions are extremely powerful for describing patterns and can be used in coordination with other tools to search and process data.

The uses and full syntax of regex far exceeds the scope of this book. You can visit the resources below for additional information and utilities related to regex.

• http://www.rexegg.com/

• https://regex101.com

• https://www.regextester.com/

• http://www.regular-expressions.info/

In the next chapter we will discuss common data types relevant to security operations and how it can be gathered.

Exercises

1. Write a regular expression that matches a floating point number (a number with a decimal point) such as 3.14. There can be digits on either side of the decimal point but there need not be any on one side or the other. Allow it to match just a decimal point by itself, too.

2. Use a back reference in a regular expression to match a number that appears on both sides of an equal sign. For example, it should match “314 is = to 314” but not “6 = 7”

3. Write a regular expression that looks for a line that begins with a digit and ends with a digit, with anything occurring in between.

4. Write a regular expression that uses grouping to match on the following 2 IP addresses: 10.0.0.25 and 10.0.0.134.

5. Write a regular expression that will match if the hexadecimal string 0x90 occurs more than 3 times in a row (i.e. 0x90 0x90 0x90).

Commands in Use

We introduce cut, file, head, and for Windows systems reg and wevtutil, to gather and select data of interest from local and remote systems.

12/05/2017 192.168.10.14 test.html
12/30/2017 192.168.10.185 login.html

$ cut -d' ' -f2 file1.txt

192.168.10.14
192.168.10.185

Pat   25
Pete  12

$ file unknownfile

unknownfile: Microsoft Word 2007+

$ reg query HKEY_LOCAL_MACHINE

HKEY_LOCAL_MACHINE\BCD00000000
HKEY_LOCAL_MACHINE\HARDWARE
HKEY_LOCAL_MACHINE\SAM
HKEY_LOCAL_MACHINE\SECURITY
HKEY_LOCAL_MACHINE\SOFTWARE
HKEY_LOCAL_MACHINE\SYSTEM

wevtutil el

wevtutil qe System //c:1 //rd:true

Gathering System Information

One of the first steps in defending a system is understanding the state of the system and what it is doing. To accomplish this you need to gather data, either locally or remotely, for analysis.

ssh myserver ps > /tmp/ps.out

ssh myserver ps \> /tmp/ps.out

ssh myserver bash < ./osdetect.sh

tar -czf ${HOSTNAME}_logs.tar.gz /var/log/

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# winlogs.sh
#
# Description:
# Gather copies of Windows log files
#
# Usage:
# winlogs.sh [-z]
#   -z Tar and zip the output
#

TGZ=0
if (( $# > 0 ))						
then
    if [[ ${1:0:2} == '-z' ]]				
    then
	TGZ=1	# tgz flag to tar/zip the log files
    fi
fi
SYSNAM=$(hostname)
LOGDIR=${1:-/tmp/${SYSNAM}_logs}			

mkdir -p $LOGDIR					

wevtutil el | while read ALOG				
do
    ALOG="${ALOG%$'\r'}"				
    echo "${ALOG}:"					
    wevtutil epl "$ALOG"  "${LOGDIR}/${SYSNAM}_${ALOG// /_}.evtx"  
done

if (( TGZ == 1 ))					  
then
    cd ${LOGDIR} && tar -czvf ${SYSNAM}_logs.tgz *.evtx   
fi

# Linux Command  |MSWin  Bash |XML tag    |Purpose
#----------------+------------+-----------+------------------------------
uname -a         |uname -a    |uname      |O.S. version etc
cat /proc/cpuinfo|systeminfo  |sysinfo    |system hardware and related info
ifconfig         |ipconfig    |nwinterface|Network interface information
route            |route print |nwroute    |routing table
arp -a           |arp -a      |nwarp      |ARP table
netstat -a       |netstat -a  |netstat    |network connections
mount            |net share   |diskinfo   |mounted disks
ps -e            |tasklist    |processes  |running processes

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# getlocal.sh
#
# Description:
# Gathers general system information and dumps it to a file
#
# Usage:
# bash getlocal.sh < cmds.txt
#   cmds.txt is a file with list of commands to run
#

# SepCmds - separate the commands from the line of input
function SepCmds()
{
      LCMD=${ALINE%%|*}                   
      REST=${ALINE#*|}                    
      WCMD=${REST%%|*}                    
      REST=${REST#*|}
      TAG=${REST%%|*}                     

      if [[ $OSTYPE == "MSWin" ]]
      then
         CMD="$WCMD"
      else
         CMD="$LCMD"
      fi
}

function DumpInfo ()
{                                                              
    printf '<systeminfo host="%s" type="%s"' "$HOSTNAME" "$OSTYPE"
    printf ' date="%s" time="%s">\n' "$(date '+%F')" "$(date '+%T')"
    readarray CMDS                           
    for ALINE in "${CMDS[@]}"                
    do
       # ignore comments
       if [[ ${ALINE:0:1} == '#' ]] ; then continue ; fi     

      SepCmds

      if [[ ${CMD:0:3} == N/A ]]             
      then
          continue
      else
          printf "<%s>\n" $TAG               
          $CMD
          printf "</%s>\n" $TAG
      fi
    done
    printf "</systeminfo>\n"
}

OSTYPE=$(./osdetect.sh)                     
HOSTNM=$(hostname)                          
TMPFILE="${HOSTNM}.info"                    

# gather the info into the tmp file; errors, too
DumpInfo  > $TMPFILE  2>&1                  

regedit //E ${HOSTNAME}_reg.bak

reg export HKEY_LOCAL_MACHINE $(uname -n)_hklm.bak

Searching the File System

The ability to search the system is critical for everything from organizing files, to incident response, to forensic investigation. The find and grep commands are extremely powerful and can be used to perform a variety of search functions.

find /home -name '*password*'

find /home -name '*password*' 2>/dev/null

find /home -name '.*'

dir c:\ /S /A:H

$ find /c -exec attrib '{}' \; | egrep '^.{4}H.*'

A   H                C:\Users\Bob\scripts\hist.txt
A   HR               C:\Users\Bob\scripts\winlogs.sh

$ find . -exec attrib '{}' \; | egrep '^.{4}H.*' | cut -c22-

C:\Users\Bob\scripts\hist.txt
C:\Users\Bob\scripts\winlogs.sh

find /home -size +5G

find / -type f -exec ls -s '{}' \; | sort -n -r | head -5

find / -type f -exec ls -s '{}' \;

!! | sort -n -r | head -5

ls / -R -s | sort -n -r | head -5

find /home -mmin -5

find /home -mtime -1

find /home -mtime +2

find /home -atime -1

find /home -type f -atime -1 -exec cp '{}' ./ \;

grep -r -i /home -e 'password'

find /home -type f -exec grep '{}' -e 'password' \; -exec cp '{}' ./ \;

$ file Title.png

Title.png: PNG image data, 366 x 84, 8-bit/color RGBA, non-interlaced

.*PNG.*

.*PNG.* 100 x 100.*

.*(PNG|JPEG).*

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# typesearch.sh
#
# Description:
# Search the file system for a given file type. It prints out the
# pathname when found.
#
# Usage:
# typesearch.sh [-c dir] [-i] [-R|r] <pattern> <path>
#   -c Copy files found to dir
#   -i Ignore case
#   -R|r Recursively search subdirectories
#   <pattern> File type pattern to search for
#   <path> Path to start search
#

DEEPORNOT="-maxdepth 1"		# just the current dir; default

# PARSE option arguments:
while getopts 'c:irR' opt; do                         
  case "${opt}" in                                    
    c) # copy found files to specified directory
	       COPY=YES
	       DESTDIR="$OPTARG"                             
	       ;;
    i) # ignore u/l case differences in search
	       CASEMATCH='-i'
	       ;;
    [Rr]) # recursive                                 
        unset DEEPORNOT;;                             
    *)  # unknown/unsupported option                  
        # error mesg will come from getopts, so just exit
        exit 2 ;;
  esac
done
shift $((OPTIND - 1))                                 


PATTERN=${1:-PDF document}                            
STARTDIR=${2:-.}	# by default start here

find $STARTDIR $DEEPORNOT -type f | while read FN     
do
    file $FN | grep -q $CASEMATCH "$PATTERN"          
    if (( $? == 0 ))   # found one                    
    then
	        echo $FN
	        if [[ $COPY ]]                               
	        then
	            cp -p $FN $DESTDIR                       
	        fi
    fi
done

find / -type f -exec file '{}' \; | egrep '.*PNG.*' | cut -d' ' -f1

This is hash file A

This is hash file B

$ sha1sum hashfilea.txt hashfileb.txt

6a07fe595f9b5b717ed7daf97b360ab231e7bbe8 *hashfilea.txt
2959e3362166c89b38d900661f5265226331782b *hashfileb..txt

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# hashsearch.sh
#
# Description:
# Recursively search a given directory for a file that
# matches a given SHA-1 hash
#
# Usage:
# hashsearch.sh <hash> <directory>
#   hash - SHA-1 hash value to file to find
#   directory - Top directory to start search
#

HASH=$1
DIR=${2:-.}	# default is here, cwd

# convert pathname into an absolute path
function mkabspath ()				
{
    if [[ $1 == /* ]]				
    then
    	ABS=$1
    else
    	ABS="$PWD/$1"				
    fi
}

find $DIR -type f |				
while read fn
do
    THISONE=$(sha1sum "$fn")			
    THISONE=${THISONE%% *}			
    if [[ $THISONE == $HASH ]]
    then
	mkabspath "$fn"				
	echo $ABS				
    fi
done

Transferring Data

Once you have gathered all of the desired data, the next step is to move it off of the origin system for further analysis. To do that you can copy the data to a removable device or upload it to a centralized server. If you are going to upload the data be sure to do so using a secure method such as Secure Copy (SCP). The example below uses scp to upload the file some_system.tar.gz to the home directory of user bob on remote system 10.0.0.45.

scp some_system.tar.gz bob@10.0.0.45:/home/bob/some_system.tar.gz

For convenience you can add a line at the end of your collection scripts to automatically use scp to upload data to a specified host. Remember to give your files unique names as to not overwrite existing files and also make analysis easier later on.

Summary

Gathering data is an important step in defensive security operations. When collecting data be sure to transfer and store it using secure methods (i.e. encrypted). As a general rule, gather all data that you think is relevant; you can easily delete data later, but you cannot analyze data you did not collect. Before collecting data, first confirm you have permission and/or legal authority to do so.

Also be aware that when dealing with adversaries, they will often try to hide their presence by deleting or obfuscating data. To counter that be sure to use multiple methods when searching for files (name, hash, contents, etc).

In the next chapter we will explore techniques for processing data and preparing it for analysis.

Exercises

1. Write the command to search the file system for any file named dog.png.

2. Write the command to search the file system for any file containing the text confidential.

3. Write the command to search the file system for any file containing the text secret or confidential and copy the file to your current working directory.

4. Write the command to execute ls -R / on the remote system 192.168.10.32 and write the output to a file named filelist.txt on your local system.

5. Modify getlocal.sh to automatically upload the results to a specified server using SCP.

6. Modify hashsearch.sh to have an option (-1) to quit after finding a match. If the option is not specified, it will keep searching for additional matches.

7. Modify hashsearch.sh to simplify the full pathname that it prints out.

   1. If the string it output was /home/usr07/subdir/./misc/x.data modify it to remove the redundant ./ before printing it out.

   2. If the string was /home/usr/07/subdir/../misc/x.data modify it to remove the ../ and also the subdir/ before printing it out.

8. Modify winlogs.sh to indicate its progress by printing the logfile name over the top of the previous logfile name. (Hint: use a return character rather than a newline)

9. Modify winlogs.sh to show a simple progress bar of plus signs building from left to right. Use a separate invocation of wevtutil el to get the count of the number of logs and scale this to, say, a width of 60.

Commands in Use

We introduce awk, join, sed, tail, and tr to prepare data for analysis.

Mike Jones
John Smith
Kathy Jones
Jane Kennedy
Tim Scott

$ awk '$2 == "Jones" {print $0}' awkusers.txt

Mike Jones
Kathy Jones

1,jdoe
2,puser
3,jsmith

0745,file1.txt,1
0830,file4.txt,2
0830,file5.txt,3

$ join -1 3 -2 1 -t, accesstime.txt usernames.txt

1,0745,file1.txt,jdoe
2,0830,file4.txt,puser
3,0830,file5.txt,jsmith

ip,OS
10.0.4.2,Windows 8
10.0.4.35,Ubuntu 16
10.0.4.107,macOS
10.0.4.145,macOS

$ sed 's/10\.0\.4\.35/10.0.4.27/g' ips.txt

ip,OS
10.0.4.2,Windows 8
10.0.4.27,Ubuntu 16
10.0.4.107,macOS
10.0.4.145,macOS

s/<regular expression>/<replace with>/<flags/

$ tail -n 1 somefile.txt

12/30/2017 192.168.10.185 login.html

tr '\\:'  '/|' < infile.txt  > outfile.txt

drive:path\name
c:\Users\Default\file.txt

drive|path/name
c|/Users/Default/file.txt

tr -d '\r' < fileWind.txt  > fileFixed.txt

$ sed -i 's/$/\r/' fileLinux.txt

Processing Delimited Files

Many of the files you will collect and process are likely to contain text, which makes the ability to manipulate text from the command line a critical skill. Text files are often broken into fields using a delimiter such as a space, tab, or comma. One of the more common formats is known as Comma Separated Values (CSV). As the name indicates, CSV files are delimited using commas, and fields may or may not be surrounded in double quotes ("). The first line of a CSV file is often the field headers. Here is an example:

"name","username","phone","password hash"
"John Smith","jsmith","555-555-1212",5f4dcc3b5aa765d61d8327deb882cf99
"Jane Smith","jnsmith","555-555-1234",e10adc3949ba59abbe56e057f20f883e
"Bill Jones","bjones","555-555-6789",d8578edf8458ce06fbc5bb76a58c5ca4

To extract just the name from the file you can use cut by specifying the field delimiter as a comma and the field number you would like returned.

$ cut -d',' -f1 csvex.txt

"name"
"John Smith"
"Jane Smith"
"Bill Jones"

Note that the field values are still enclosed in double quotations. This may not be desirable for certain applications. To remove the quotations you can simply pipe the output into tr with its -d option.

$ cut -d',' -f1 csvex.txt | tr -d '"'

name
John Smith
Jane Smith
Bill Jones

You can further process the data by removing the field header using the tail command’s -n option.

$ cut -d',' -f1 csvex.txt | tr -d '"' | tail -n +2

John Smith
Jane Smith
Bill Jones

The -n +2 option tells tail to output the contents of the file starting at line number 2, thus removing the field header.

"name","username","phone","password hash"
"John Smith","jsmith","555-555-1212",5f4dcc3b5aa765d61d8327deb882cf99
"Jane Smith","jnsmith","555-555-1234",e10adc3949ba59abbe56e057f20f883e
"Bill Jones","bjones","555-555-6789",d8578edf8458ce06fbc5bb76a58c5ca4

password,md5hash
123456,e10adc3949ba59abbe56e057f20f883e
password,5f4dcc3b5aa765d61d8327deb882cf99
welcome,40be4e59b9a2a2b5dffb918c0e86b3d7
ninja,3899dcbab79f92af727c2190bbd8abc5
abc123,e99a18c428cb38d5f260853678922e03
123456789,25f9e794323b453885f5181f1b624d0b
12345678,25d55ad283aa400af464c76d713c07ad
sunshine,0571749e2ac330a7455809c6b0e7af90
princess,8afa847f50a716e64932d995c8e7435a
qwerty,d8578edf8458ce06fbc5bb76a58c5c

$ awk -F "," '{print $4}' csvex.txt

"password hash"
5f4dcc3b5aa765d61d8327deb882cf99
e10adc3949ba59abbe56e057f20f883e
d8578edf8458ce06fbc5bb76a58c5ca4

$ grep "$(awk -F "," '{print $4}' csvex.txt)" passwords.txt

123456,e10adc3949ba59abbe56e057f20f883e
password,5f4dcc3b5aa765d61d8327deb882cf99
qwerty,d8578edf8458ce06fbc5bb76a58c5ca4

$ cut -d',' -f3 csvex.txt | cut -c2-13 | tail -n +2

555-555-1212
555-555-1234
555-555-6789

Processing XML

Extensible Markup Language (XML) allows you to arbitrarily create tags and elements that describe data. Below is an example XML document.

<book title="Rapid Cybersecurity Ops" edition="1">
  <author>
    <firstName>Paul</firstName>
    <lastName>Troncone</lastName>
  </author>
  <author>
    <firstName>Carl</firstName>
    <lastName>Albing</lastName>
  </author>
</book>

This is a start tag that contains two attributes, also known as name/value pairs. Attribute values must always be quoted.

This is a start tag.

This is an element that has content.

This is an end tag.

For useful processing, you must be able to search through the XML and extract data from within the tags, which can be done using grep. Lets find all of the firstName elements. The -o option is used so only the text that matches the regex pattern will be returned, rather than the entire line.

$ grep -o '<firstName>.*<\/firstName>' book.xml

<firstName>Paul</firstName>
<firstName>Carl</firstName>

Note that the regex pattern above will only find the XML element if the start and end tags are on the same line. To find the pattern across multiple lines you need to make use of two special features. First, add the -z option to grep, which treats newlines like any ordinary character in its searching and adds a null (ASCII 0) at the end of each string it finds. Then add the -P option and (?s) to the regex pattern, which is a Perl-specific pattern match modifier. It modifies the . metacharacter to also match on the newline character.

$ grep -Pzo '(?s)<author>.*?<\/author>' book.xml

<author>
  <firstName>Paul</firstName>
  <lastName>Troncone</lastName>
</author><author>
  <firstName>Carl</firstName>
  <lastName>Albing</lastName>
</author>

To strip the XML start and end tags and extract the content you can pipe your output into sed.

$ grep -Po '<firstName>.*?<\/firstName>' book.xml | sed 's/<[^>]*>//g'

Paul
Carl

The sed expression can be described as s/expr/other/ to replace (or substitute) some expression (expr) with something else (other). The expression can be just literal characters or a more complex regex. If an expression has no “other” portion, such as s/expr// then it replaces anything that matches the regular expression with nothing, essentially removing it. The regex pattern we use in the above example, namely the <[^>]*> expression, is a little confusing, so lets break it down.

< - The pattern begins with a literal less-than character <

[^>]* - Zero or more (indicated by the asterisk) characters from the set of characters inside the brackets; the first character is a ^ which means “not” any of the remaining characters listed. Here that’s just the solitary greater-than character, so [^>] matches any character that is not >

> - The pattern ends with a literal >

This should match a single XML tag, from its opening less-than to its closing greater-than character, but not more than that.

Processing JSON

JavaScript Object Notation (JSON) is another popular file format, particularly for exchanging data through Application Programming Interfaces (APIs). JSON is a simple format that consists of objects, arrays, and name/value pairs. Here is a sample JSON file:

{
  "title": "Rapid Cybersecurity Ops",
  "edition": 1,
  "authors": [
    {
      "firstName": "Paul",
      "lastName": "Troncone"
    },
    {
      "firstName": "Carl",
      "lastName": "Albing"
    }
  ]
}

This is an object. Objects begin with { and end with }.

This is a name/value pair. Values can be a string, number, array, boolean, or null.

This is an array. Arrays begin with [ and end with ].

When processing JSON you are likely going to want to extract key/value pairs. To do that you can use grep. Lets extract the firstName key/value pair from book.json.

$ grep -o '"firstName": ".*"' book.json

"firstName": "Paul"
"firstName": "Carl"

Again, the -o option is used to return only the characters that match the pattern rather than the entire line of the file.

If you want to remove the key and only display the value you can do so by piping the output into cut, extracting the second field, and removing the quotations with tr.

$ grep -o '"firstName": ".*"' book.json | cut -d " " -f2 | tr -d '\"'

Paul
Carl

We will perform more advanced processing of JSON in a later chapter.

Aggregating Data

Data is often collected from a variety of sources, and in a variety of files and formats. Before you can analyze the data you must get it all into the same place and in a format that is conducive to analysis.

Suppose you want to search a treasure trove of data files for any system named ProductionWebServer. Recall that in previous scripts we wrapped our collected data in XML tags with the following format: '<systeminfo host="">. During collection we also named our files using the host name. You can now use either of those attributes to find and aggregate the data into a single location.

find /data -type f -exec grep '{}' -e 'ProductionWebServer' \;
-exec cat '{}' >> ProductionWebServerAgg.txt \;

The command find /data -type f lists all of the files in the /data directory and its subdirectories. For each file found, it runs grep looking for the string ProductionWebServer. If found, the file is appended (>>) to the file ProductionWebServerAgg.txt. Replace the cat command with cp and a directory location if you would rather copy all of the files to a single location rather than to a single file.

You can also use the join command to take data that is spread across two files and aggregate it into one. Take the two files seen in Example 5-10 and Example 5-11.

ip,OS
10.0.4.2,Windows 8
10.0.4.35,Ubuntu 16
10.0.4.107,macOS
10.0.4.145,macOS

user,ip
jdoe,10.0.4.2
jsmith,10.0.4.35
msmith,10.0.4.107
tjones,10.0.4.145

The files share a common column of data, which is the IP addresses. Because of that the files can be merged using join.

$ join -t, -2 2 ips.txt user.txt

ip,OS,user
10.0.4.2,Windows 8,jdoe
10.0.4.35,Ubuntu 16,jsmith
10.0.4.107,macOS,msmith
10.0.4.145,macOS,tjones

The -t, option tells join that the columns are delimited using a comma, by default it uses a space character.

The -2 2 option tells join to use the second column of data in the second file (user.txt) as the key to perform the merge. By default join uses the first field as the key, which is appropriate for the first file (ips.txt). If you needed to join using a different field in ips.txt you would just add the option -1 n where n is replaced by the appropriate column number.

Summary

In this chapter we explored ways to process common data formats including delimited, positional, JSON, and XML. The vast majority of data you collect and process will be in one of those formats.

In the next chapter we will look at how data can be analyzed and transformed into information that will provide insights into system status and drive decision making.

Exercises

1. Given the file tasks.txt below, use the cut command to extract columns 1 (Image Name), 2 (PID), and 5 (Mem Usage).

   Image Name;PID;Session Name;Session#;Mem Usage
   System Idle Process;0;Services;0;4 K
   System;4;Services;0;2,140 K
   smss.exe;340;Services;0;1,060 K
   csrss.exe;528;Services;0;4,756 K

2. Given the file procowner.txt below, use the join command to merge the file with tasks.txt.

   Process Owner;PID
   jdoe;0
   tjones;4
   jsmith;340
   msmith;528

3. Use the tr command to replace all of the semicolon characters in tasks.txt with the tab character and print it to the screen.

4. Write a command that extracts the first and last names of all of the authors in book.json.

Commands in use

We introduce sort, head, and uniq to limit the data we need to process and display. The following file will be used for command examples:

12/05/2017 192.168.10.14 test.html
12/30/2017 192.168.10.185 login.html

sort -k 2 file1.txt

sort -k 1.5,1.7 file1.txt

Sorting and Arranging Data

When analyzing data for the first time it is often beneficial to start by looking at the extremes; the things that occurred the most or least frequently, the smallest or largest data transfers, etc. For example, consider the data you can collect from web server log files. An unusually high number of page accesses could indicate scanning activity or a denial of service attempt. An unusually high number of bytes downloaded by a host could indicate site cloning or data exfiltration.

To do that you can use the sort, head, and tail commands at the end of a pipeline such as:

…   | sort -k 2.1 -rn | head -15

which pipes the output of a script into the sort command and then pipes that sorted output into head that will print the top 15 (in this case) lines. The sort command here is using as its sort key (-k) the second field beginning at its first character (2.1). Moreover, it is doing a reverse sort (-r) and the values will be sorted like numbers (-n). Why a numerical sort? so that 2 shows up between 1 and 3 and not between 19 and 20 (which is alphabetical order).

By using head we take the first lines of the output. We could get the last few lines by piping the output from the sort command into tail instead of head. Using tail -15 would give us the last 15 lines. The other way to do this would be to simply remove the -r option on sort so that it does an ascending rather than descending sort.

Counting Occurrences in Data

A typical web server log can contain tens of thousands of entries. By counting each time a page was accessed, or by which IP address it was accessed from you can gain a better understanding of general site activity. Interesting entries can include:

• A high number of requests returning the 404 (Page Not Found) status code for a specific page; this can indicate broken hyperlinks.

• A high number of requests from a single IP address returning the 404 status code; this can indicate probing activity looking for hidden or unlinked pages.

• A high number of requests returning the 401 (Unauthorized) status code, particularly from the same IP address; this can indicate an attempt at bypassing authentication, such as brute-force password guessing.

To detect this type of activity we need to be able to extract key fields, such as the source IP address, and count the number of times they appear in a file. To accomplish this we will use the cut command to extract the field and then pipe the output into our new tool countem.sh.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# countem.sh
#
# Description:
# Count the number of instances of an item using bash
#
# Usage:
# countem.sh < inputfile
#

declare -A cnt        # assoc. array             
while read id xtra                               
do
    let cnt[$id]++                               
done
# now display what we counted
# for each key in the (key, value) assoc. array
for id in "${!cnt[@]}"                           
do
    printf '%d %s\n'  "${cnt[$id]}"  "$id"       
done

And here is another version, this time using awk:

# Rapid Cybersecurity Ops
# countem.awk
#
# Description:
# Count the number of instances of an item using awk
#
# Usage:
# countem.awk < inputfile
#

awk '{ cnt[$1]++ }
END { for (id in cnt) {
        printf "%d %s\n", cnt[id], id
      }
    }'

Since we don’t know what IP addresses (or other strings) we might encounter, we will use an associative array, declared here with the -A option, so that we can use whatever string we read as our index.

The associative array feature of bash found in bash 4.0 and higher. In such an array, the index doesn’t have to be a number but can be any string. So you can index the array by the IP address and thus count the occurrences of that IP address. In case you’re using something older than bash 4.0, Example 6-4 is an alternate script that uses awk instead.

The array references are like others in bash, using the ${var[index]} syntax to reference an element of the array. To get all the different index values that have been used (the “keys” if you think of these arrays as (key, value) pairings), use: ${!cnt[@]}

While we only expect one word of input per line, we put the variable xtra there to capture any other words that appear on the line. Each variable on a read command gets a assigned the corresponding word from the input (i.e., the first variable gets the first word, the second variable get the second word, and so on), but the last variable gets any and all remaining words. On the other hand, if there are fewer words of input on a line than their are variables on the read command, then those extra variables get set to the empty string. So for our purposes, if there are extra words on the input line, they’ll all be assigned to xtra but if there are no extra words then xtra will be given the value of the null string (which won’t matter either way because we don’t use it.)

Here we use that string as the index and increment its previous value. For the first use of the index, the previous value will be unset, which will be taken as zero.

This syntax lets us iterate over all the various index values that we encountered. Note, however, that the order is not guaranteed -it has to do with the hashing algorithm for the index values, so it is not guaranteed to be in any order such as alphabetical order.

In printing out the value and key we put the values inside quotes so that we always get a single value for each argument - even if that value had a space or two inside it. It isn’t expected to happen with our use of this script, but such coding practices make the scripts more robust when used in other situations.

Both will work nicely in a pipeline of commands like this:

cut -d' ' -f1 logfile | bash countem.sh

or (see note 2 above) just:

bash countem.sh < logfile

For example, to count the number of times an IP address made a HTTP request that resulted in a 404 (page not found) error:

$ awk '$9 == 404 {print $1}' access.log | bash countem.sh

1 192.168.0.36
2 192.168.0.37
1 192.168.0.11

You can also use grep 404 access.log and pipe it into countem.sh, but that would include lines where 404 appears in other places (e.g. the byte count, or part of a file path). The use of awk here restricts the counting only to lines where the returned status (the ninth field) is 404. It then prints just the IP address (field 1) and pipes the output into countem.sh to get the total number of times each IP address made a request that resulted in a 404 error.

To begin analysis of the example access.log file you can start by looking at the hosts that accessed the web server. You can use the Linux cut command to extract the first field of the log file, which contains the source IP address, and then pipe the output into the countem.sh script. The exact command and output is seen below.

$ cut -d' ' -f1 access.log | bash countem.sh | sort -rn

111 192.168.0.37
55 192.168.0.36
51 192.168.0.11
42 192.168.0.14
28 192.168.0.26

$ cut -d' ' -f1 access.log | sort | uniq -c | sort -rn

111 192.168.0.37
55 192.168.0.36
51 192.168.0.11
42 192.168.0.14
28 192.168.0.26

Next, you can further investigate by looking at the host that had the most number of requests, which as can be seen above is IP address 192.168.0.37 with 111. You can use awk to filter on the IP address, then pipe that into cut to extract the field that contains the request, and finally pipe that output into countem.sh to provide the total number of requests for each page.

$ awk '$1 == "192.168.0.37" {print $0}' access.log | cut -d' ' -f7 | bash countem.sh

1 /uploads/2/9/1/4/29147191/31549414299.png?457
14 /files/theme/mobile49c2.js?1490908488
1 /cdn2.editmysite.com/images/editor/theme-background/stock/iPad.html
1 /uploads/2/9/1/4/29147191/2992005_orig.jpg
. . .
14 /files/theme/custom49c2.js?1490908488

The activity of this particular host is unimpressive, appearing to be standard web browsing behavior. If you take a look at the host with the next highest number of requests, you will see something a little more interesting.

$ awk '$1 == "192.168.0.36" {print $0}' access.log | cut -d' ' -f7 | bash countem.sh

1 /files/theme/mobile49c2.js?1490908488
1 /uploads/2/9/1/4/29147191/31549414299.png?457
1 /_/cdn2.editmysite.com/.../Coffee.html
1 /_/cdn2.editmysite.com/.../iPad.html
. . .
1 /uploads/2/9/1/4/29147191/601239_orig.png

This output indicates that host 192.168.0.36 accessed nearly every page on the website exactly one time. This type of activity often indicates webcrawler or site cloning activity. If you take a look at the user agent string provided by the client it further verifies this conclusion.

$ awk '$1 == "192.168.0.36" {print $0}' access.log | cut -d' ' -f12-17 | uniq

"Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)

The user agent identifies itself as HTTrack, which is a tool used to download or clone websites. While not necessarily malicious, it is interesting to note during analysis.

Totaling Numbers in Data

Rather than just count the number of times an IP address or other item occurs, what if you wanted to know the total byte count that has been sent to an IP address - or which IP addresses have requested and received the most data?

The solution is not that much different than countem.sh - you just need a few small changes. First, you need more columns of data by tweaking the input filter (the cut command) to extract two columns (IP address and byte count) rather than just IP address. Second, you will change the calculation from an increment, (let cnt[$id]++) a simple count, to be a summing of that second field of data (let cnt[$id]+=$data).

The pipeline to invoke this will now extract two fields from the logfile, the first and the last.

cut -d' ' -f 1,10 access.log | bash summer.sh

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# summer.sh
#
# Description:
# Sum the total of field 2 values for each unique field 1
#
# Usage:
# Input Format - <input field> <number>
#

declare -A cnt        # assoc. array
while read id count
do
  let cnt[$id]+=$count
done
for id in "${!cnt[@]}"
do
    printf "%-15s %8d\n"  "${id}"  "${cnt[${id}]}" 
done

Note that we’ve made a few other changes to the output format. With the output format, we’ve added field sizes of 15 characters for the first string (the IP address in our sample data), left justified (via the minus sign) and 8 digits for the sum values. If the sum is larger, it will print the larger number, and if the string is longer, it will be printed in full. We’ve done this to get the data to align, by and large, nicely in columns, for readability.

You can run summer.sh against the example access.log file to get an idea of the total amount of data requested by each host. To do this use cut to extract the IP address and bytes transferred fields, and then pipe the output into summer.sh.

$ cut -d' ' -f1,10 access.log | bash summer.sh | sort -k 2.1 -rn

192.168.0.36     4371198
192.168.0.37     2575030
192.168.0.11     2537662
192.168.0.14     2876088
192.168.0.26      665693

These results can be useful in identifying hosts that have transferred unusually large amounts of data compared to other hosts. A spike could indicate data theft and exfiltration. If you identify such a host the next step would be to review the specific pages and files accessed by the suspicious host to try and classify it as malicious or benign.

Displaying Data in a Histogram

You can take counting one step further by providing a more visual display of the results. You can take the output from countem.sh or summer.sh and pipe it into yet another script, one that will produce a histogram-like display of the results.

The script to do the printing will take the first field as the index to an associative array; the second field as the value for that array element. It will then iterate through the array and print a number of hashtags to represent the count, scaled to 50 # symbols for the largest count in the list.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# histogram.sh
#
# Description:
# Generate a horizontal bar chart of specified data
#
# Usage:
# Data input format - <label> <value>
#

function pr_bar ()                            
{
    local -i i raw maxraw scaled              
    raw=$1
    maxraw=$2
    ((scaled=(MAXBAR*raw)/maxraw))            
    # min size guarantee
    ((raw > 0 && scaled == 0)) && scaled=1				

    for((i=0; i<scaled; i++)) ; do printf '#' ; done
    printf '\n'

} # pr_bar

#
# "main"
#
declare -A RA						
declare -i MAXBAR max
max=0
MAXBAR=50	# how large the largest bar should be

while read labl val
do
    let RA[$labl]=$val					
    # keep the largest value; for scaling
    (( val > max )) && max=$val
done

# scale and print it
for labl in "${!RA[@]}"					
do
    printf '%-20.20s  ' "$labl"
    pr_bar ${RA[$labl]} $max				
done

We define a function to draw a single bar of the histogram. This definition must be encountered before a call to the function can be made, so it makes sense to put function definitions at the front of our script. We will be reusing this function in a future script so we could have put it in a separate file and included it here with a source command - but we didn’t.

We declare all these variables as local because we don’t want them to interfere with variable names in the rest of this script (or any others, if we copy/paste this script to use elsewhere). We declare all these variables as integer (that’s the -i option) because we are only going to compute values with them and not use them as strings.

The computation is done inside double-parentheses and inside those we don’t need to use the $ to indicate “the value of” each variable name.

This is an “if-less” if statement. If the expression inside the double-parentheses is true then, and only then, is the second expression (the assignment) executed. This will guarantee that scaled is never zero when the raw value is non-zero. Why? Because we’d like something to show up in that case.

The main part of the script begins with a declaration of the RA array as an associative array.

Here we reference the associative array using the label, a string, as its index.

Since the array isn’t index by numbers, we can’t just count integers and use them as indices. This contruct gives all the various strings that were used as an index to the array, one at a time, in the for loop.

We use the label as an index one more time to get the count and pass it as the first parameter to our pr_bar function.

Note that the items don’t appear in the same order as the input. That’s because the hashing algorithm for the key (the index) doesn’t preserve ordering. You could take this output and pipe it into yet another sort, or you could take a slightly different approach.

Here’s a version of the histogram script that preserves order - by not using an associative array. This might also be useful on older versions of bash (pre 4.0), prior to the introduction of associative arrays. Only the “main” part of the script is shown as the function pr_bar remains the same.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# histogram_plain.sh
#
# Description:
# Generate a horizontal bar chart of specified data without
# using associative arrays, good for older versions of bash
#
# Usage:
# Data input format - <label> <value>
#

declare -a RA_key RA_val                                 
declare -i max ndx
max=0
maxbar=50    # how large the largest bar should be

ndx=0
while read labl val
do
    RA_key[$ndx]=$labl                                   
    RA_value[$ndx]=$val
    # keep the largest value; for scaling
    (( val > max )) && max=$val
    let ndx++
done

# scale and print it
for ((j=0; j<ndx; j++))                                  
do
    printf "%-20.20s  " ${RA_key[$j]}
    pr_bar ${RA_value[$j]} $max
done

This version of the script avoids the use of associative arrays - in case you are running an older version of bash (prior to 4.x), such as on MacOS systems. For this version we use two separate arrays, one for the index value and one for the counts. Since they are normal arrays we have to use an integer index and so we will keep a simple count in the variable ndx.

Here the variable names are declared as arrays. The lower-case a says that they are arrays, but not of the “associative” variety. While not strictly necessary, it is good practice.

The key and value pairs are stored in separate arrays, but at the same index location. This approach is “brittle” - that is, easily broken, if changes to the script ever got the two arrays out of sync.

Now the for loop, unlike the previous script, is a simple counting of an integer from 0 to ndx. The variable j is used here so as not to interfere with the index in the for looop inside pr_bar although we were careful enough inside the function to declare its version of i as local to the function. Do you trust it? Change the j to an i here and see if it still works (It does). Then try removing the local declaration and see if it fails (It does).

This approach with the two arrays does have one advantage. By using the numerical index for storing the label and the data you can retrieve them in order they were read in - in the numerical order of the index.

You can now visually see the hosts that transferred the largest number of bytes by extracting the appropriate fields from access.log, piping the results into summer.sh and then into histogram.sh.

$ cut -d' ' -f1,10 access.log | bash summer.sh | bash histogram.sh

192.168.0.36          ##################################################
192.168.0.37          #############################
192.168.0.11          #############################
192.168.0.14          ################################
192.168.0.26          #######

While this might not seem that useful for the small amount of sample data, being able to visualize trends is invaluable when looking across larger datasets.

In addition to looking at the number of bytes transferred by IP address or host, it is often interesting to look at the data by date and time. To do that you can use the summer.sh script, but due to the format of the access.log file you need to do a little more processing before you can pipe it into the script. If you use cut to extract the date/time and bytes transferred fields you are left with data that causes some problems for the script.

$ cut -d' ' -f4,10 access.log

[12/Nov/2017:15:52:59 2377
[12/Nov/2017:15:52:59 4529
[12/Nov/2017:15:52:59 1112

As seen in the output above, the raw data starts with a [ character. That causes a problem with the script because it denotes the beginning of an array in bash. To remedy that you can use an additional iteration of the cut command to remove the character using -c2- as an option. This option tells cut to extract the data by character starting at position 2 and going to the end of the line (-). The corrected output with the square bracket removed can be seen below.

$ cut -d' ' -f4,10 access.log | cut -c2-

12/Nov/2017:15:52:59 2377
12/Nov/2017:15:52:59 4529
12/Nov/2017:15:52:59 1112

cut -d' ' -f4,10 access.log | tr -d '['

You also need to determine how you want to group the time-bound data; by day, month, year, hour, etc. You can do this by simply modifying the option for the second cut iteration. The table below illustrates the cut option to use to extract various forms of the date/time field. Note that these cut options are specific to Apache log files.

The histogram.sh script can be particularly useful when looking at time-based data. For example, if your organization has an internal web server that is only accessed during working hours of 9:00 AM to 5:00 PM, you can review the server log file on a daily basis using the histogram view and see if there are any spikes in activity outside of normal working hours. Large spikes of activity or data transfer outside of normal working hours could indicate exfiltration by a malicious actor. If any anomalies are detected you can filter the data by that particular date and time and review the page accesses to determine if the activity is malicious.

For example, if you want to see a histogram of the total amount of data that was retrieved on a certain day and on an hourly basis you can do the following:

$ awk '$4 ~ "12/Nov/2017" {print $0}' access.log | cut -d' ' -f4,10 |
cut -c14-15,22- | bash summer.sh | bash histogram.sh

17              ##
16              ###########
15              ############
19              ##
18              ##################################################

Here the access.log file is sent through awk to extract the entries from a particular date. Note the use of the like operator (~) in stead of == since field 4 also contains time information. Those entries are piped into cut to extract the date/time and bytes transferred fields, and then piped into cut again to extract just the hour. From there it is summed by hour using summer.sh and converted into a histogram using histogram.sh. The result is a histogram that displays the total number of bytes transferred each hour on November 12, 2017.

Finding Uniqueness in Data

Previously IP address 192.168.0.37 was identified as the system that had the largest number of page requests. The next logical question is what pages did this system request? With that answer you can start to gain an understanding of what the system was doing on the server and categorize the activity as benign, suspicious, or malicious. To accomplish that you can use awk and cut and pipe the output into countem.sh.

$ awk '$1 == "192.168.0.37" {print $0}' access.log | cut -d' ' -f7 |
bash countem.sh | sort -rn | head -5

14 /files/theme/plugin49c2.js?1490908488
14 /files/theme/mobile49c2.js?1490908488
14 /files/theme/custom49c2.js?1490908488
14 /files/main_styleaf0e.css?1509483497
3 /consulting.html

While this can be accomplished by piping together commands and scripts, that requires multiple passes through the data. This may work for many datasets, but it is too inefficient for extremely large datasets. You can streamline this by writing a bash script specifically designed to extract and count page accesses, and only requires a single pass over the data.

# Rapid Cybersecurity Ops
# pagereq.sh
#
# Description:
# Count the number of page requests for a given IP address using bash
#
# Usage:
# pagereq <ip address> < inputfile
#   <ip address> IP address to search for
#

declare -A cnt                                             
while read addr d1 d2 datim gmtoff getr page therest
do
    if [[ $1 == $addr ]] ; then let cnt[$page]+=1 ; fi
done
for id in ${!cnt[@]}                                       
do
    printf "%8d %s\n" ${cnt[$id]} $id
done

We declare cnt as an associative array (also known as a hash table or dictionary) so that we can use a string as the index to the array. In this program we will be using the page address (the URL) as the index.

The ${!cnt[@]} results in a list of all the different index values that have been encountered. Note, however, that they are not listed in any useful order.

Early versions of bash don’t have associative arrays. You can use awk to do the same thing - counting the various page requests from a particular ip address - since awk has associative arrays.

# Rapid Cybersecurity Ops
# pagereq.awk
#
# Description:
# Count the number of page requests for a given IP address using awk
#
# Usage:
# pagereq <ip address> < inputfile
#   <ip address> IP address to search for
#

# count the number of page requests from an address ($1)
awk -v page="$1" '{ if ($1==page) {cnt[$7]+=1 } }                
END { for (id in cnt) {                                          
    printf "%8d %s\n", cnt[id], id
    }
}'

There are two very different $1 variables on this line. The first $1 is a shell variable and refers to the first argument supplied to this script when it is invoked. The second $1 is an awk variable. It refers to the first field of the input on each line. The first $1 has been assigned to the awk variable page so that it can be compared to each $1 of awk - that is, to each first field of the input data.

This simple syntax results in the varialbe id iterating over the values of the index values to the cnt array. It is much simpler syntax than the shell’s "${!cnt[@]}" syntax, but with the same effect.

You can run pagereq.sh by providing the IP address you would like to search for and redirect access.log as input.

$ bash pagereq.sh 192.168.0.37 < access.log | sort -rn | head -5

14 /files/theme/plugin49c2.js?1490908488
14 /files/theme/mobile49c2.js?1490908488
14 /files/theme/custom49c2.js?1490908488
14 /files/main_styleaf0e.css?1509483497
3 /consulting.html

Identifying Anomalies in Data

On the web a User Agent String is a small piece of textual information sent by a browser to a web server that identifies the client’s operating system, browser type, version, and other information. It is typically used by web servers to ensure page compatibility with the user’s browser. Here is an example of a user agent string:

Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0

This user agent string identifies the system as: Windows NT version 6.3 (aka Windows 8.1); 64-bit architecture; and using the Firefox browser.

The user agent string is interesting for a few reasons: first because of the significant amount of information it conveys, which, can be used to identify the types of systems and browsers accessing the server; second because it is configurable by the end user, which, can be used to identify systems that may not be using a standard browser or may not be using a browser at all (i.e. a webcrawler).

You can identify unusual user agents by first compiling a list of known good user agents. For the purposes of this exercise we will use a very small list that is not specific to a particular version.

Firefox
Chrome
Safari
Edge

You can then read in a web server log and compare each line to each valid user agent until you get a match. If no match is found it should be considered an anomaly and printed to standard out along with the IP address of the system making the request. This provides yet another vantage point into the data, identifying systems with unusual user agents, and another path to further explore.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# useragents.sh
#
# Description:
# Read through a log looking for unknown user agents
#
# Usage:
# useragents.txt < <inputfile>
#   <inputfile> Apache access log
#


# mismatch - search through the array of known names
#  returns 1 (false) if it finds a match
#  returns 0 (true) if there is no match
function mismatch ()                                    
{
    local -i i                                          
    for ((i=0; i<$KNSIZE; i++))
    do
        [[ "$1" =~ .*${KNOWN[$i]}.* ]] && return 1      
    done
    return 0
}

# read up the known ones
readarray -t KNOWN < "useragents.txt"                      
KNSIZE=${#KNOWN[@]}                                     

# preprocess logfile (stdin) to pick out ipaddr and user agent
awk -F'"' '{print $1, $6}' | \
while read ipaddr dash1 dash2 dtstamp delta useragent   
do
    if mismatch "$useragent"
    then
        echo "anomaly: $ipaddr $useragent"
    fi
done

We will use a function for the core of this script. It will return a success (or “true”) if it finds a mismatch, that is, if it finds no match against the list of known user agents. This logic may seem a bit inverted, but it makes the if statement containing the call to mismatch read clearly.

Declaring our for loop index as a local variable is good practice. It’s not strictly necessary in this script but is a good habit.

There are two strings to compare - the input from the logfile and a line from the list of known user agents. To make for a very flexible comparison we use the regex comparison operator (the =~). The .* (meaning “zero or more instances of any character”) placed on either side of the $KNOWN array reference means that the known string can appear anywhere within the other string for a match.

Each line of the file is added as an element to the array name specified. This gives us an array of known user agents. There are two identical ways to do this in bash either readarray, as used here, or mapfile. The -t option removes the trailing newline from each line read. The file containing the list of known user agents is specified here; modify as needed.

This computes the size of the array. It is used inside the mismatch function to loop through the array. We calculate it here, once, outside our loop to avoid recomputing it every time the function is called.

The input string is a complex mix of words and quote marks. To capture the user agent string we use the double-quote as the field separator. Doing that, however, means that our first field contains more than just the ip address. By using the bash read we can parse on the spaces to get the ip address. The last argument of the read takes all the remaining words and so it can capture all the several words of the user agent string.

Summary

In this chapter we looked at techniques to analyze the content of log files by identifying unusual and anomolous activity. This type of analysis can provide you with insights into what occurred in the past. In the next chapter we will look at how to analyze log files and other data to provide insights into what is happening in the system in real time.

Exercises

1. The example use of summer.sh used cut to print the 1st and 10th fields of the access.log file, like this:

$ cut -d' ' -f1,10 access.log | bash summer.sh | sort -k 2.1 -rn

Replace the cut command by using the awk command. Do you get the same results? What might be different about those two approaches?

1. Expand the histogram.sh script to include the count at the end of each histogram bar. Here is sample output:

   192.168.0.37          #############################    2575030
   192.168.0.26          ####### 665693

2. Expand the histogram.sh script to allow the user to supply the option -s that specifies the maximum bar size. For example histogram.sh -s 25 would limit the maximum bar size to 25 # characters. The default should remain at 50 if no option is given.

3. Download the following web log file TODO: Add Log File URL.

   1. Which IP address made the most number of requests?

   2. Which page was accessed the most number of times?

4. Download the following Domain Name System (DNS) server log TODO: Add Log File URL

   1. What was the most requested domain?

   2. What day had the most number of requests?

5. Modify the useragents.sh script to add some parameters

   1. Add code for an optional first parameter to be a filename of the known hosts. If not specified, default to the name known.hosts as it currently is used.

   2. Add code for a -f option to take an argument. The argument is the filename of the logfile to read rather than reading from stdin.

6. Modify the pagereq.sh script to not need an associative array but to work with a traditional array that uses a numerical index. Convert the ip address into a 10-12 digit number for that use. Caution: don’t have leading zeros on the number or the shell will attempt to interpret it as an octal number. Example: convert “10.124.16.3” into “10124016003” which can be used as a numerical index.

Monitoring Text Logs

The most basic method to monitor a log in real time is to use the tail command’s -f option, which continuously reads a file and outputs new lines to stdout as they are added. As in previous chapters, we will use an Apache web server access log for examples, but the techniques presented can be applied to any text-based log. To monitor the Apache access log with tail:

tail -f /var/logs/apache2/access.log

Commands can be combined to provide more advanced functionality. The output from tail can be piped into grep so only entries matching specific criteria will be output. The example below monitors the Apache access log and outputs entries matching a particular IP address.

tail -f /var/logs/apache2/access.log | grep '10.0.0.152'

Regular expressions can also be used. Below only entries returning a HTTP status code of 404 Page Not Found will be displayed. The -i option is added to ignore character case.

tail -f /var/logs/apache2/access.log | egrep -i 'HTTP/.*" 404'

To clean up the output it can be piped into the cut command to remove extraneous information. The example below monitors the access log for requests resulting in a 404 status code and then uses cut to only display the date/time and the page that was requested.

$ tail -f access.log | egrep --line-buffered 'HTTP/.*" 404' | cut -d' ' -f4-7

[29/Jul/2018:13:10:05 -0400] "GET /test
[29/Jul/2018:13:16:17 -0400] "GET /test.txt
[29/Jul/2018:13:17:37 -0400] "GET /favicon.ico

You can further clean the output by piping it into tr -d '[]"' to remove the square brackets and the orphen double-quotation.

Note that we used the egrep command’s --line-buffered option. This forces egrep to output to stdout each time a line break occurs. Without this option buffering occurs and output is not piped into cut until a buffer is filled. We don’t want to wait that long. The option will have egrep write out each line as it finds it.

\.\./ 
etc/passwd 
etc/shadow
cmd\.exe 
/bin/sh
/bin/bash

tail -f /var/logs/apache2/access.log | egrep -i -f ioc.txt

tail -f /var/logs/apache2/access.log | egrep --line-buffered -i -f ioc.txt |
tee -a interesting.txt

Monitoring Windows Logs

As previously discussed, you need to use the wevtutil command to access Windows events. While the command is very versatile, it does not have functionality similar to tail that can be used to extract new entries as they occur. Thankfully, a simple bash script can provide similar functionality.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# wintail.sh
#
# Description:
# Perform a tail-like function on a Windows log
#

WINLOG="Application"  

LASTLOG=$(wevtutil qe "$WINLOG" //c:1 //rd:true //f:text)  

while true
do
	CURRENTLOG=$(wevtutil qe "$WINLOG" //c:1 //rd:true //f:text)  
	if [[ "$CURRENTLOG" != "$LASTLOG" ]]
	then
		echo "$CURRENTLOG"
		echo "----------------------------------"
		LASTLOG="$CURRENTLOG"
	fi
done

This variable identifies the Windows log you want to monitor. You can use wevtutil el to obtain a list of logs currently available on the system.

This executes the wevtutil command to query the specified log file. The c:1 parameter causes it to return only one log entry. The rd:true parameter causes the command to read the most recent log entry. Finally, f:text returns the result as plain text rather than XML which makes it easy to read from the screen.

The next few lines execute the wevtutil command again and compare the latest log entry to the last one printed to the screen. If the two are different, meaning that a new entry was added to the log, it prints the entry to the screen. If they are the same nothing happens and it loops back and checks again.

Generating a Real-Time Histogram

A tail -f provides an ongoing stream of data. What if you wanted to count how many lines are added to a file during a time interval? You could observe that stream of data, start a timer, and begin counting until a specified time interval is up; then you can stop counting and report the results.

You might divide this work into two separate processes - two separate scripts - one to count the lines and another to watch the clock. The timekeeper will notify the line counter by means of a standard POSIX inter-process communication mechanism called a “signal”. A signal is a software interrupt and there are different kinds of such interrupts. Some are fatal, that is, will cause the process to terminate (e.g., a floating point exception). Most can be ignored or caught - and an action taken when the signal is caught. Many have a predefined purpose, used by the operating system. We’ll use one of the two signals available for users, SIGUSR1. (The other is SIGUSR2.)

Shell scripts can catch the catchable interrupts with the trap command, a shell built-in command. With trap you specify a command to indicate what action you want taken and a list of signals which trigger the invocation of that command. For example:

trap warnmsg SIGINT

will cause the command warnmsg (our own script or function) to be called whenever the shell script receives a SIGINT signal, as when you type a control-C to interrupt a running process.

Here is the script that performs the count.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# looper.sh
#
# Description:
# Count the lines in a file being tailed -f
# Report the count interval on every SIGUSR1
#


function interval ()					
{
    echo $(date '+%y%m%d %H%M%S') $cnt			
    cnt=0
}

declare -i cnt=0
trap interval SIGUSR1					

shopt -s lastpipe					

tail -f --pid=$$ ${1:-log.file} | while read aline	
do
    let cnt++
done

The function interval will be called on each signal. We define it here. It needs to be defined before we can call it, of course, but also before we can use it in our trap statement, below.

The date command is called to provide a timestamp for the count value that we print out. After we print the count we reset its value to 0 to start the count for the next interval.

Now that interval is defined, we can tell bash to call the function whenever our process receives a SIGUSR1 signal.

This is a crucial step. Normally when there is a pipeline of commands (such as ls -l | grep rwx | wc) then those pieces of the pipeline (each command) are run in subshells and they each end up with their own process id. This would be a problem for this script because the while loop would be in a subshell, with a different process id. Whatever process started the looper.sh script wouldn’t know the process id of the while loop to send the signal to it. Moreover, changing the value of the cnt variable in the subshell doesn’t change the value of cnt in the main process, so a signal to the main process would result in a value of zero every time. The solution is this shopt command that sets (-s) the shell option lastpipe. That option tells the shell not to create a subshell for the last command in a pipeline but to run that command in the same process as the script itself. In our case that means that the tail will run in a subshell (i.e., a different process) but the while loop will be part of the main script process. Caution: this shell option is only available in bash 4.x and above, and is only for non-interactive shells (i.e., scripts).

Here is the tail -f command with one more option, the --pid option. We specify a process id to tell tail to exit when that process dies. We are specifying $$, the current shell script’s process id, as the one to watch. This is useful for cleanup so that we don’t get tail commands left running in the background (if, for example, this script is run in the background; see the next script which does just that.)

The script tailcount.sh starts and stops the counting, the script that has the “stopwatch” so to speak, and times these intervals.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# tailcount.sh
#
# Description:
# Count lines every n seconds
#

# cleanup - the other processes on exit
function cleanup ()
{
  [[ -n $LOPID ]] && kill $LOPID		
}

trap cleanup EXIT 				

bash looper.sh $1 &				
LOPID=$!					
# give it a chance to start up
sleep 3

while true
do
    kill -SIGUSR1 $LOPID
    sleep 5
done >&2					

Since this script will be starting other processes (other scripts) it should clean up after itself. If the process id has been stored in LOPID the variable will be non-empty and therefore the function will send a signal via the kill command to that process. By not specifying a particular signal on the kill command, the default signal to be sent is SIGTERM.

Not a signal, EXIT is a special case for the trap statement to tell the shell to call this function (here, cleanup) when the shell that is running this script is about to exit.

Now the real work begins. The looper.sh script is called but is put in the “background”, that is, detached from the keyboard to run on its own while this script continues (without waiting for looper.sh to finish).

This saves the process id of the script that we just put in the background.

This redirection is just a precaution. By redirecting stdout into stderr then any and all output coming from the while loop or the kill or sleep statements (though we’re not expecting any) will be sent to stderr and not get mixed in with any output coming from looper.sh which, though it is in the background, still writes to stdout.

In summary, looper.sh has been put in the background and its process id saved in a shell variable. Every 5 seconds this script (tailcount.sh) sends that process (which is running looper.sh) a SIGUSR1 signal which causes looper.sh to print out its current count and restart its counting. When tailcount.sh exits it will clean up by sending a SIGTERM to the looper.sh function so that it, too, will be terminated.

With both a script to do the counting and a script to drive it with its “stopwatch”, you can use their output as input to a script that prints out a histogram-like bar to represent the count. It is invoked as follows:

bash tailcount.sh | bash livebar.sh

The livebar.sh script reads from stdin and prints its output to stdout, one line for each line of input.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# livebar.sh
#
# Description:
# Creates a rolling horizontal bar chart of live data
#
# Usage:
# <output> | bash livebar.sh
#

function pr_bar ()					
{
    local raw maxraw scaled
    raw=$1
    maxraw=$2
    ((scaled=(maxbar*raw)/maxraw))
    ((scaled == 0)) && scaled=1		# min size guarantee
    for((i=0; i<scaled; i++)) ; do printf '#' ; done
    printf '\n'

} # pr_bar


maxbar=60   # largest no. of chars in a bar		
MAX=60
while read dayst timst qty
do
    if (( qty > MAX ))					
    then
	let MAX=$qty+$qty/4	# allow some room
	echo "              **** rescaling: MAX=$MAX"
    fi
    printf '%6.6s %6.6s %4d:' $dayst $timst $qty	
    pr_bar $qty $MAX
done

The pr_bar function prints the bar of hashtags scaled to the maximum size based on the parameters supplied. This function might look familiar. We’re using the same function we used in histogram.sh in the previous chapter.

This is the longest string of hastags we will allow on a line (to avoid line wrap).

How large will the values be that need to be displayed? Not knowing before hand (although it could be supplied as an argument to the script) the script will, instead, keep track of a maximum. If that maximum is exceeded it will “rescale” and the current and future lines will be scaled to the new maximum. The script adds 25% onto the maximum so that it doesn’t need to rescale if each new value goes up by just one or two each time.

The printf specifies a min and max width on the first two fields that are printed. They are date and time stamps and will be truncated if they exceed those widths. You wouldn’t want the count truncated so we specify it will be 4 digits wide but the entire value will be printed regardless. If it is smaller than 4 it will be padded with blanks.

Since this script reads from stdin you can run it by itself to see how it behaves. Here’s a sample:

$ bash livebar.sh
201010 1020 20
201010   1020   20:####################
201010 1020 70
              **** rescaling: MAX=87
201010   1020   70:################################################
201010 1020 75
201010   1020   75:###################################################
^C

In this example the input is mixing with the output. You could also put the input into a file and redirect it into the script to see just the output.

$ bash livebar.sh < testdata.txt
bash livebar.sh < x.data
201010   1020   20:####################
              **** rescaling: MAX=87
201010   1020   70:################################################
201010   1020   75:###################################################
$

Summary

Log files can provide tremendous insight into the operation of a system, but they also come in large quantities which makes them challenging to analyze. You can minimize this issue by creating a series of scripts to automate data formatting, aggregation, and alerting.

In the next chapter we will look at how similar techniques can be leveraged to monitor networks for configuration changes.

Exercises

1. add a -i option to livebar.sh to set the interval in seconds.

2. add a -M option to livebar.sh to set an expected maximum for input values. Use the getopts builtin to parse your options.

3. How might you add a -f option that filters data using (using, e.g., grep)? What challenges might you encounter? What approach(es) might you take to deal with those?

4. Modify wintail.sh to allow the user to specify the Windows log to be monitored by passing in a command line argument.

5. Modify wintail.sh to add the capability for it to be a lightweight intrusion detection system using egrep and an IOC file.

6. Consider the statement made in the note about buffering: “When the input is coming from a file, that usually happens quickly.” Why “usually”? Under what conditions might you see the need for the line buffering option on grep even when reading from a file?

Commands and Arguments

The basic operation of bash is to execute a command, that is, to run another program. When several words appear on the command line bash assumes that the first word is the name of the program to run and the remaining words are the arguments to the command. For example:

mkdir -p /tmp/scratch/garble

will have bash run the command called mkdir and it will pass it two arguments -p and /tmp/scratch/garble. By convention programs generally put their options first, and have them begin with a leading "-“, as is the case here with the -p option. This particular command will create a directory called /tmp/scratch/garble. The -p option will mean that no errors will be reported and any intervening directories will be created (or attempted) as needed (e.g., if only /tmp exists, it will create /tmp/scratch before attempting to create /tmp/scratch/garble).

Standard Input/Output/Error

A running program is called a process and every process in the Unix/Linux/Posix (and thus Windows) environment has three distinct input/output file descriptors. These three are called “standard input” (or stdin, for short), “standard output” (stdout), and “standard error” (stderr).

As you might guess by its name, stdin is the default source for input to a program, by default the characters coming from the keyboard. When your script reads from stdin it is reading characters typed on the keyboard or (as we shall see shortly) it can be changed to read from a file. Stdout is the default place for sending output from a program. By default the output appears in the window which is running your shell or shell script. Standard error can also be sent output from a program, but it is (or should be) where error messages are written. It’s up to the person writing the program to direct any output to either stdout or stderr. So be conscientious when writing your scripts to send any error messages not to stdout but to stderr (as shown below).

Redirection and Piping

One of the great innovations of the shell was that it gave you a mechanism whereby you could take a running program and change where it got its input and/or change where it sent its output without modifying the program itself. If you have a program called handywork and it reads its input from stdin and writes its results to stdout, then you can change its behavior as simply as this:

handywork < data.in  > results.out

which will run handywork but will have the input come not from the keyboard but instead from the data file called data.in (assuming such a file exists and has input in the format we want). Similarly the output is being sent not to the screen but into a file called results.out (which will be created if it doesn’t exist and overwritten if it does). This technique is called “redirection” because we are re-directing input to come from a different place and re-directing output to go somewhere other than the screen.

What about stderr? The syntax is similar. We have to distinguish between stout and stderr when redirecting data coming out of the program and we make this distinction through the use of the file descriptor numbers. Stdin is file descriptor 0, stdout is file descriptor 1 and stderr is file descriptor 2 so we can redirect error messages this way:

handywork 2> err.msgs

which will redirect only stderr and send any such error message output to a file we called err.msgs (for obvious reasons).

Of course we can do all three on the same line:

handywork < data.in  > results.out  2> err.msgs

Sometimes we want the error messages combined with the normal output (as it does by default when both are written to the screen). We can do this with the following syntax:

handywork < data.in  > results.out 2>&1

which says to send stderr (2) to the same location as file descriptor 1 (”&1“). Note that without the ampersand, the error messages would just be sent to a file named “1”. This combining of stdout and stderr is so common that there is a useful shorthand notation:

handywork < data.in  &> results.out

If you want to discard standard output you can redirect it to a special file called /dev/null as follows:

handywork < data.in > /dev/null

To view output on the command line and simultaneously redirect that same output to a file, use the tee command. The following will display the output of handywork to the screen and also save it to results.out:

handywork < data.in | tee results.out

A file will be created or truncated (i.e., contents discarded) when output is redirected. If you want to preserve the file’s existing content you can, instead, append to the file using a double greater-than sign, like this:

handywork < data.in  >> results.out

This will execute handywork and then any output from stdout will be appended to the file results.out rather than overwriting its existing content.

Similarly this command line:

handywork < data.in  &>> results.out

will execute handywork and then append both stdout and stderr to the file results.out rather than overwriting its existing content.

Running Commands in the Background

Throughout this book we will be going beyond one-line commands and will be building complex scripts. Some of these scripts can take a significant amount of time to execute, so much so that you may not want to spend time waiting for them to complete. Instead, you can run any command or script in the background using the & operator. The script will continue to run, but you can issue other commands and/or run other scripts. For example, to run ping in the background and redirect standard output to a file:

ping 192.168.10.56 > ping.log &

You will likely want to redirect standard output and/or standard error to a file when sending tasks to the background, or the task will continue to print to the screen and interrupt other activities you are performing.

You can use the jobs command to list any tasks currently running in the background.

$ jobs
[1]+  Running                 ping 192.168.10.56 > ping.log &

Use the fg command and the corresponding job number to bring the task back into the foreground.

$ fg 1
ping 192.168.10.56 > ping.log

If your task is currently executing in the foreground you can use CTRL-Z to suspend the process and then bg to continue the process in the background. From there you can use jobs and fg as described above.

From Command Line to Script

A shell script is just a file that contains the same commands that you could type on the command line. Put one or more commands into a file and you have a shell script. If you called your file myscript you can run that script by typing: bash myscript or you can give it “execute permission” (e.g., chmod 755 myscript) and then you can invoke it directly: ./myscript to run the script. We often include the following line as the first line of the script, which tells the operating system which scripting language we are using:

#!/bin/bash -

Of course this assumes that bash is located in the /bin directory. If your script needs to be more portable, you could use this approach instead:

#!/usr/bin/env bash

It uses the env command to look up the location of bash and is considered the standard way to address the portability problem. It makes the assumption, however, that the env command is to be found in /usr/bin.

Summary

In this chapter you saw how to run single commands and redirect input and output. In the next chapter we will discuss the real power of scripting, which comes from being able to run commands repeatedly, make decisions in the script, and loop over a variety of inputs.

Exercises

1. Write a command that executes ifconfig and redirects standard output to a file named ipaddress.txt.

2. Write a command that executes ifconfig and redirects standard output and appends it to a file named ipaddress.txt.

3. Write a command that copies all of the files in the directory /etc/a to the directory /etc/b and redirects standard error to the file copyerror.log.

4. Write a command that performs a directory listing (ls) on the root file directory and pipes the output into the more command.

5. Write a command that executes mytask.sh and sends it to the background.

6. Given the job list below, write the command that brings the Amazon ping task to the foreground.

   [1]   Running                 ping www.google.com > /dev/null &
   [2]-  Running                 ping www.amazon.com > /dev/null &
   [3]+  Running                 ping www.oreilly.com > /dev/null &

Output

As with any programming language, bash has the ability to output information to the screen. Output can be achieved using the echo command.

$ echo "Hello World"

Hello World

You may also use the printf command which allows for some additonal formatting.

$ printf "Hello World"

Hello World

Variables

Bash variables begin with an alphabetic character or underscore followed by alphanumeric characters. They are string variables unless declared otherwise. To assign a value to the variable, you write something like this:

MYVAR=textforavalue

To retrieve the value of that variable, for example to print out the value using the echo command, you use the $ in front of the variable name, like this:

echo $MYVAR

If you want to assign a series of words to the variable, that is, to preserve any whitespace, use quotation marks around the value, as in:

MYVAR='here is a longer set of words'
OTHRV="either double or single quotes will work"

The use of double quotes will allow other substitutions to occur inside the string. For example:

firstvar=beginning
secondvr="this is just the $firstvar"
echo $secondvr

This will result in the output: this is just the beginning

There are a variety of substitutions that can occur when retrieving the value of a variable; we will show those as we use them in the scripts to follow.

You can also store the output of a shell command using $( ) as follows:

CMDOUT=$(pwd)

That will execute the command pwd in a sub-shell and rather than printing the the result to stdout, it will store the output of the command in the variable CMDOUT. You can also pipe together multiple commands within the $ ( ).

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# echoparams.sh
#
# Description:
# Demonstrates accessing parameters in bash
#
# Usage:
# ./echoparms.sh <param 1> <param 2> <param 3>
#

echo $#
echo $0
echo $1
echo $2
echo $3

$ ./echoparams.sh bash is fun

3
./echoparams.sh
bash
is
fun

Input

User input is received in bash using the read command. The read command will obtain user input from the command line and store it in a specified variable. The script below reads user input into the MYVAR variable and then prints it to the screen.

read MYVAR
echo "$MYVAR"

Conditionals

Bash has a rich variety of conditionals. Many, but not all, begin with the keyword if.

Any command or program that you invoke in bash may do some output but it will also always return a success or fail value. In the shell this value can be found in the $? variable immediately after a command has run. A return value of 0 is considered “success” or “true”; any non-zero value is considered “error” or “false”. The simplest form of the if statement uses this fact. It takes the form:

if cmd
then
   other cmds
fi

For example, the script below attempts to change directories to /tmp. If that command is successful (returns 0) the body of the if statement will execute.

if  cd /tmp
then
    echo "here is what is in /tmp:"
    ls -l
fi

Bash can even handle a pipeline of commands in a similar fashion:

if ls | grep pdf
then
    echo found one or more pdf files here
fi

With a pipeline, it is the success/failure of the last command in the pipeline that determines if the “true” branch is taken. Here is an example where that fact matters:

ls | grep pdf | wc

This series of commands will be “true” even if no pdf string is found by the grep command. That is because the wc command (a word count of the input) will print:

0       0       0

That output indicates 0 characters, 0 words, and 0 lines when no output comes from the grep command. That is still a successful (or true) result, not an error or failure. It counted as many lines as it was given, even if it was given zero lines to count.

A more typical form of if used for comparisons makes use of the compound command [[ or the shell built-in commands [ or test. Use these to test file attributes or to make comparisons of value.

To test if a file exists on the file system:

if [[ -e $FILENAME ]]
then
    echo $FILENAME exists
fi

Table 2-1 lists additional tests that can be done on files using if comparisons.

To test if the variable $VAL is less than the variable $MIN:

if [[ $VAL -lt $MIN ]]
then
    echo "value is too small"
fi

Table 2-2 lists additional numeric tests that can be done using if comparisons.

if [[ $VAL < $OTHR ]]

If you want to do numerical comparisons with the less-than sign, use the double parentheses construct. It assumes that the variables are all numerical and will evaluate them as such. Empty or unset variables are evaluated as 0. Inside the parentheses you don’t need the $ operator to retrieve a value, except for positional parameters like $1 and $2 (so as not to confuse them with the constants 1 and 2). For example:

if (( VAL < 12 ))
then
    echo "value $VAL is too small"
fi

In bash you can even make branching decisions without an explicit if/then construct. Commands are typically separated by a newline - that is, they appear one per line. You can get the same effect by separating them with a semicolon. If you write cd $DIR ; ls then bash will perform the cd and then the ls.

Two commands can also be separated by either && or || symbols. If you write cd $DIR && ls then the ls command will run only if the cd command succeeds. Similarly if you write cd $DIR || echo cd failed the message will be printed only if the cd fails.

You can use the [[ syntax to make various tests, even without an explicit if.

[[ -d $DIR ]] && ls "$DIR"

means the same as if you had written

if [[ -d $DIR ]]
then
  ls "$DIR"
fi

[[ -d $DIR ]] || echo "error: no such directory: $DIR" ; exit

[[ -d $DIR ]] || { echo "error: no such directory: $DIR" ; exit ; }

Looping

Looping with a while statement is similar to the if construct in that it can take a single command or a pipeline of commands for the decision of true or false. It can also make use of the brackets or parentheses as in the if examples, above.

In some languages braces ( { } ) are used to group the statement together that are the body of the while loop. In others, like python, indentation is the indication of which statements are the loop body. In bash, however, the statements are grouped between two keywords: do and done.

Here is a simple while loop:

i=0
while (( i < 1000 ))
do
    echo $i
    let i++
done

The loop above will execute while the variable i is less than 1000. Each time the body of the loop executes it will print the value of i to the screen. It then uses the let command to execute i++ as an arithmetic expression, thus incrementing i by 1 each time.

Here is a more complicated while loop that executes commands as part of its condition.

while ls | grep -q pdf
do
    echo -n 'there is a file with pdf in its name here: '
    pwd
    cd ..
done

A for loop is also available in bash - in three variations.

Simple numerical looping can be done using the double parentheses construct. It looks much like the for loop in C or Java, but with double parentheses and with do and done instead of braces:

for ((i=0; i < 100; i++))
do
    echo $i
done

Another useful form of the for loop is used to iterate through all the parameters that are passed to a shell script (or function within the script), that is, $1, $2, $3 an so on. Note that ARG in args.sh can be replaced with any variable name of your choice.

for ARG
do
    echo here is an argument: $ARG
done

Here is the output of args.sh when three parameters are passed in.

$ ./args.sh bash is fun

here is an argument: bash
here is an argument: is
here is an argument: fun

Finally, for an arbitrary list of values, use a similar form of the for statement simply naming each of the values you want for each iteration of the loop. That list can be explicitly written out, like this:

for VAL in 20 3 dog peach 7 vanilla
do
    echo $VAL
done

The values used in the for loop can also be generated by calling other programs or using other shell features:

for VAL in $(ls | grep pdf) {0..5}
do
    echo $VAL
done

Here the variable VAL will take, in turn, the value for each of the filenames that ls piped into grep finds with the letters pdf in its filename (e.g. “doc.pdf” or “notapdfile.txt”) and then each of the numbers 0 through 5. It may not be that sensible to have the variable VAL be a filename sometimes and a single digit another time, but this shows you that it can be done.

Functions

Define a function with sytnax like this:

function myfun ()
{
  # body of the function goes here
}

Not all that syntax is necessary - you can use either "function" or "()" - you don’t need both. We recommend, and will be using, both - mostly for readability.

There are a few important considerations to keep in mind with bash functions:

• Unless declared with the local builtin command inside the function, variables are global in scope. A for loop which sets and increments i could be messing with the value of i used elsewhere in your code.

• The braces are the most commonly used grouping for the function body, but any of the shell’s compound command syntax is allowed - though why, e.g., would you want the function to run in a sub-shell?

• Redirecting I/O on the braces does so for all the statements inside the function. Examples of this will be seen in upcomoing chapters.

• No parameters are declared in the function definition. Whatever and however many arguments are supplied on the invocation of the function are passed to it.

The function is called (invoked) just like any command is called in the shell. Having defined myfun as a function you can call it like this:

myfun 2 /arb "14 years"

which calls the function myfun supplying it with 3 arguments.

Pattern Matching in bash

When you need to name a lot of files on a command line, you don’t need to type each and every name. Bash provides pattern matching (sometimes called “wildcarding”) to allow you to specify a set of files with a pattern. The easiest one is simply an asterisk * (or “star”) which will match any number of any characters. When used by itself, therefore, it matches all files in the current directory. The asterisk can be used in conjunction with other characters. For example \*.txt matches all the files in the current directory which end with the four characters .txt. The pattern /usr/bin/g\* will match all the files in /usr/bin that begin with the letter g.

Another special character in pattern matching is ? the question mark, which matches a single character. For example, source.? will match source.c or source.o but not source.py or source.cpp.

The last of the three special pattern matching characters are [ ], the square brackets. A match can be made with any one of the characters listed inside the square brackets, so the pattern x[abc]y matches any or all of the files named xay, xby, or xcy, assuming they exist. You can specify a range within the square brackets, like [0-9] for all digits. If the first character within the brackets is either a \! or a ^ then the pattern means anything other than the remaining characters in the brackets. For example, [aeiou] would match a vowel whereas [^aeiou] would match any character except the vowels (including digits and punctuation characters).

Similar to ranges, you can specify character classes within braces. Table 2-3 lists the character classes and their description.

Character classes are specified like this: [:cntrl:] within square brackets (so you have two sets of []). For example, this pattern: \*[[:punct:]]jpg will match any filename that has any number of any characters followed by a punctuation character followed by the letters jpg. So it would match files named wow!jpg or some,jpg or photo.jpg but not a file named this.is.myjpg since there is no punctuation character right before the jpg.

There are more complex aspects of pattern matching if you turn on the shell option extglob (like this: shopt -s extglob) so that you can repeat patterns or negate patterns. We won’t need these in our example scripts but we encourage you to learn about them (e.g., via the bash man page).

There are a few things to keep in mind when using shell pattern matching:

• Patterns aren’t regular expressions (discussed later); don’t confuse the two.

• Patterns are matched against files in the file system; if the pattern begins with a pathname (e.g., /usr/lib ) then the matching will be done against files in that directory.

• If no pattern is matched, the shell will use the special pattern matching characters as literal characters of the filename; for example, if your script says echo data > /tmp/*.out but there is no file in /tmp that ends in .out then the shell will create a file called *.out in the /tmp directory. Remove it like this: rm /tmp/\*.out by using the backslash to tell the shell not to pattern match with the asterisk.

• No pattern matching occurs inside of quotes (either double or single quotes), so if your script says echo data > "/tmp/*.out" it will create a file called /tmp/*.out (which we recommend you avoid doing).

Writing Your First Script - Detecting Operating System Type

Now that we have gone over the fundamentals of the command line and bash you are ready to write your first script. The bash shell is available on a variety of platforms including Linux, Windows, macOS, and Git Bash. As you write more complex scripts in the future it is imperative that you know what operating system you are interacting with as each one has a slightly different set of commands available. The osdetect.sh script helps you in making that determination.

The general idea of the script is that it will look for a command that is unique to a particular operating system. The limitation is that on any given system an administrator may have created and added a command with that name, so this is not foolproof.

#!/bin/bash -
#
# Rapid Cybersecurity Ops
# osdetect.sh
#
# Description:
# Distinguish between MS-Windows/Linux/MacOS
#
# Usage:
# Output will be one of: Linux MSWin macOS
#

if type -t wevtutil &> /dev/null           
then
    OS=MSWin
elif type -t scutil &> /dev/null           
then
    OS=macOS
else
    OS=Linux
fi
echo $OS

We use the type built-in in bash to tell us what kind of a command (alias, keyword, function, built-in, or file) its arguments are. The -t option tells it to print nothing if the command isn’t found. The command returns as “false” in that case. We redirect all the output (both stdout and stderr) to /dev/null thereby throwing it away, as we only want to know if the wevtutil command was found.

Again we use the type built-in but this time we are looking for the scutil command which is available on macOS systems.

Summary

The bash shell can be seen as a programming language, one with variables and if/then/else statements, loops, and functions. It has its own syntax, similar in many ways to other programming languages, but just different enough to catch you if you’re not careful.

It has its strengths - like easily invoking other programs or connecting sequences of other programs - and it has its weaknesses: it doesn’t have floating point arithmetic or much support (though some) for complex data structures.

In the chapters ahead we will describe and use many bash features and OS commands in the context of cybersecurity operations. We will further explore some of the features we have touched on here, and other more advanced or obsure features. Keep your eyes out for those featues and practice and use them for your own scripting.

Exercises

1. Experiment with the uname command, seeing what it prints on the various operating systems. Re-write the osdetect.sh script to use the uname command, possibly with one of its options. Caution: not all options are available on every operating system.

2. Modify the osdetect.sh script to use a function. Put the if/then/else logic inside the function and then call it from the script. Don’t have the function itself do any output. Make the output come from the main part of the script.

3. Set the permissions on the osdetect.sh script to be executable (see man chmod) so that you can run the script without using bash as the first word on the command line. How do you now invoke the script?

4. Write a script called argcnt.sh that tells how many arguments are supplied to the script.

   1. Modify your script to have it also echo each argument one per line.

   2. Modify your script further to label each argument like this:

      $ bash argcnt.sh this is a "real live" test
      there are 5 arguments
      arg1: this
      arg2: is
      arg3: a
      arg4: real live
      arg5: test
      $

5. Modify argcnt.sh so it only lists the even arguments.

Commands in Use

We introduce the grep family of commands to demonstrate the basic regex patterns.

grep -R -i 'password' /home

Regular Expression Metacharacters

Regular expressions are patterns that are created using a series of characters and metacharacters. Metacharacters such as "?" and "*" have special meaning beyond their literal meaning in regex.

$ grep 'T.o' frost.txt

1    Two roads diverged in a yellow wood,

$ egrep 'T.?o' frost.txt

1    Two roads diverged in a yellow wood,
5    To where it bent in the undergrowth;

$ grep 'T.*o' frost.txt

1    Two roads diverged in a yellow wood,
5    To where it bent in the undergrowth;
7 Excerpt from The Road Not Taken by Robert Frost

$ egrep 'T.+o' frost.txt

1    Two roads diverged in a yellow wood,
5    To where it bent in the undergrowth;
7 Excerpt from The Road Not Taken by Robert Frost

$ egrep 'And be one (stranger|traveler), long I stood' frost.txt

3    And be one traveler, long I stood

$ grep -P '\d' frost.txt

1    Two roads diverged in a yellow wood,
2    And sorry I could not travel both
3    And be one traveler, long I stood
4    And looked down one as far as I could
5    To where it bent in the undergrowth;
6
7 Excerpt from The Road Not Taken by Robert Frost

grep 'X[[:upper:][:digit:]]' idlist.txt

User: XTjohnson
an XWing model 7
an X7wing model

1    Command
2    <i>line</i>
3    is
4    <div>great</div>
5    <u>!</u>

$ egrep '<([A-Za-z]*)>.*</\1>' tags.txt

2    <i>line</i>
4    <div>great</div>
5    <u>!</u>

Summary

Regular expressions are extremely powerful for describing patterns and can be used in coordination with other tools to search and process data.

The uses and full syntax of regex far exceeds the scope of this book. You can visit the resources below for additional information and utilities related to regex.

• http://www.rexegg.com/

• https://regex101.com

• https://www.regextester.com/

• http://www.regular-expressions.info/

In the next chapter we will discuss common data types relevant to security operations and how it can be gathered.

Exercises

1. Write a regular expression that matches a floating point number (a number with a decimal point) such as 3.14. There can be digits on either side of the decimal point but there need not be any on one side or the other. Allow it to match just a decimal point by itself, too.

2. Use a back reference in a regular expression to match a number that appears on both sides of an equal sign. For example, it should match “314 is = to 314” but not “6 = 7”

3. Write a regular expression that looks for a line that begins with a digit and ends with a digit, with anything occurring in between.

4. Write a regular expression that uses grouping to match on the following 2 IP addresses: 10.0.0.25 and 10.0.0.134.

5. Write a regular expression that will match if the hexadecimal string 0x90 occurs more than 3 times in a row (i.e. 0x90 0x90 0x90).