The term obfuscation refers to a process of obscuring meaningful information. Malware authors often use various obfuscation techniques to hide the information and to modify the malicious content to make detection and analysis difficult for a security analyst. Adversaries typically use encoding/encryption techniques to conceal the information from the security products. In addition to using encoding/encryption, an attacker uses a program such as packers to obfuscate the malicious binary content, which makes analysis and reverse engineering much more difficult. In this chapter, we will look at identifying these obfuscation techniques and how to decode/decrypt and unpack the malicious binaries. We will begin by looking at the encoding/encryption techniques, and later we will look at the unpacking techniques.

Adversaries typically use encoding and encryption for the following reasons:

• To conceal command and control communication
• To hide from a signature-based solution such as Intrusion prevention systems
• To obscure the content of the configuration file used by the malware
• To encrypt information to be exfiltrated from the victim system
• To obfuscate strings in the malicious binary to hide from static analysis

Before we delve into how malware uses an encryption algorithm, let's try to understand the basics and some of the terms that we will use throughout this chapter. A plaintext refers to an unencrypted message; this might be a command and control (C2) traffic or content of the file that malware wants to encrypt. A ciphertext refers to an encrypted message; this might be an encrypted executable or encrypted command that malware receives from the C2 server.

Malware encrypts the plaintext, by passing it as input along with the key to an encryption function, which produces a ciphertext. The resultant ciphertext is typically used by the malware to write to file or send over the network:

In the same manner, malware may receive an encrypted content from the C2 server or from the file and then decrypt it by passing the encrypted content and the key to the decryption function, as follows:

While analyzing malware, you may want to understand how a particular content is encrypted or decrypted. To do this, you will mainly focus on identifying either the encryption or the decryption function and the key used to encrypt or decrypt the content. For instance, if you wish to determine how the network content is encrypted, then you will likely find the encryption function just before the network output operation (such as HttpSendRequest()). In the same manner, if you wish to know how the encrypted content from the C2 is decrypted, then you are likely to find the decryption function after the content is retrieved from C2 using an API such as InternetReadFile().

Once the encryption/decryption function is identified, examining these functions will give you an idea as to how the content is encrypted/decrypted, the key used, and the algorithm used to obfuscate the data.