In this chapter, we are going to explore the basic techniques and motivations for obfuscation of Linux programs. Techniques that obfuscate or encrypt binaries or make them difficult to tamper with are called software protection schemes. By "software protection," we mean binary protection or binary hardening techniques. Binary hardening is not exclusive to Linux; in fact, there are many more products for the Windows OS in this technology genre, and there are definitely more examples to choose from for discussion.
What many people fail to realize is that Linux has a market for this too, although it largely exists for anti-tamper products used by the government. There are also a number of ELF binary protectors that were released over the last decade in the hacker community, several of which paved the way for many of the technologies used today.
An entire book could be dedicated to the art of software protection, and as the author of some of the more recent binary protection technologies for ELF, I could easily get carried away with this chapter. Instead, I will stick to explaining the fundamentals and some interesting techniques that are used, followed by some insights into my own binary protector—Maya's Veil. The tricky engineering and skills that go into binary protection make it a challenging topic to articulate, but I will do my best here.
A packer is a type of software that is commonly used by malware authors and hackers to compress or encrypt an executable in order to obfuscate its code and data. One very common packer is named UPX (http://upx.sourceforge.net) and is available as a package on most Linux distributions. The original purpose of this type of packer was to compress an executable and make it smaller.
Since the code is compressed, it must have a way to decompress itself before executing in memory—this is where things get interesting, and we will discuss how this works in the Stub mechanics and the userland exec section. At any rate, malware authors have realized that compressing their malware-infected files would evade AV detection due to obfuscation. This led malware/antivirus researchers to develop automated unpackers, which are now used in most, if not all, modern AV products.
Nowadays, the term "packed binary" refers not only to compressed binaries but also to encrypted binaries or binaries that are shielded with an obfuscation layer of any kind. Since the early 2000s, there have been several remarkable ELF binary protectors that have shaped the future of binary protection in Linux. We will explore each one of these and use them to model the different techniques used to protect ELF binaries. Beforehand, however, let's look at how stubs work to load and execute a compressed or encrypted binary.