Table of Contents for
Learn Linux Shell Scripting - Fundamentals of Bash 4.4

Version ebook / Retour

Cover image for bash Cookbook, 2nd Edition Learn Linux Shell Scripting - Fundamentals of Bash 4.4 by Sebastiaan Tammer Published by Packt Publishing, 2018
  1. Learn Linux Shell Scripting - Fundamentals of Shell 4.4
  2. Title Page
  3. Copyright and Credits
  4. Learn Linux Shell Scripting – Fundamentals of Bash 4.4
  5. About Packt
  6. Why subscribe?
  7. PacktPub.com
  8. Contributors
  9. About the author
  10. About the reviewer
  11. Packt is searching for authors like you
  12. Table of Contents
  13. Preface
  14. Who this book is for
  15. What this book covers
  16. To get the most out of this book
  17. Download the example code files
  18. Download the color images
  19. Conventions used
  20. Get in touch
  21. Reviews
  22. Disclaimer
  23. Introduction
  24. What is Linux?
  25. What is Bash?
  26. Summary
  27. Setting Up Your Local Environment
  28. Technical requirements
  29. Choosing between a virtual machine and a physical installation
  30. Setting up VirtualBox
  31. Creating an Ubuntu virtual machine
  32. Creating the virtual machine in VirtualBox
  33. Installing Ubuntu on the virtual machine
  34. Accessing the virtual machine via SSH
  35. Summary
  36. Questions
  37. Further reading
  38. Choosing the Right Tools
  39. Technical requirements
  40. Using graphical editors for shell scripting
  41. Atom
  42. Atom installation and configuration
  43. Notepad++
  44. Using command-line editors
  45. Vim
  46. Vim summary
  47. .vimrc
  48. Vim cheat sheet
  49. nano
  50. Combining graphical editors with command-line editors when writing shell scripts
  51. Summary
  52. Questions
  53. Further reading
  54. The Linux Filesystem
  55. Technical requirements
  56. The Linux filesystem explained
  57. What is a filesystem?
  58. What makes the Linux filesystem unique?
  59. Structure of the Linux filesystem
  60. Tree structure
  61. Overview of top-level directories
  62. What about multiple partitions?
  63. /bin/, /sbin/, and /usr/
  64. /etc/
  65. /opt/, /tmp/, and /var/
  66. Everything is a file
  67. Different types of files
  68. Summary
  69. Questions
  70. Further reading
  71. Understanding the Linux Permissions Scheme
  72. Technical requirements
  73. Read, write, and execute
  74. RWX
  75. Users, groups, and others
  76. Manipulating file permissions and ownership
  77. chmod, umask
  78. sudo, chown, and chgrp
  79. sudo
  80. chown, chgrp
  81. Working with multiple users
  82. Advanced permissions
  83. File attributes
  84. Special file permissions
  85. Access Control Lists (ACLs)
  86. Summary
  87. Questions
  88. Further reading
  89. File Manipulation
  90. Technical requirements
  91. Common file operations
  92. Copying
  93. Removing
  94. Renaming, moving, and linking
  95. Archiving
  96. Finding files
  97. locate
  98. find
  99. Summary
  100. Questions
  101. Further reading
  102. Hello World!
  103. Technical requirements
  104. First steps
  105. The shebang
  106. Running scripts
  107. Readability
  108. Comments
  109. Script header
  110. Verbosity
  111. Verbosity in comments
  112. Verbosity of commands
  113. Verbosity of command output
  114. Keep It Simple, Stupid (KISS)
  115. Summary
  116. Questions
  117. Further reading
  118. Variables and User Input
  119. Technical requirements
  120. What is a variable?
  121. Why do we need variables?
  122. Variables or constants?
  123. Variable naming
  124. Dealing with user input
  125. Basic input
  126. Parameters and arguments
  127. Interactive versus non-interactive scripts
  128. Combining positional arguments and read
  129. Summary
  130. Questions
  131. Further reading
  132. Error Checking and Handling
  133. Technical requirements
  134. Error checking
  135. Exit status
  136. Functional checks
  137. Test shorthand
  138. Variable refresher
  139. Bash debugging
  140. Error handling
  141. if-then-exit
  142. if-then-else
  143. Shorthand syntax
  144. Error prevention
  145. Checking arguments
  146. Managing absolute and relative paths
  147. Dealing with y/n
  148. Summary
  149. Questions
  150. Further reading
  151. Regular Expressions
  152. Technical requirements
  153. Introducing regular expressions
  154. What is a regular expression?
  155. grep
  156. Greediness
  157. Character matching
  158. Line anchors
  159. Character classes
  160. Globbing
  161. What is globbing?
  162. Similarities with regular expressions
  163. More globbing
  164. Advanced globbing
  165. Disabling globbing, and other options
  166. Using regular expressions with egrep and sed
  167. Advanced grep
  168. Introducing egrep
  169. sed, the stream editor
  170. Stream editing
  171. In-place editing
  172. Line manipulation
  173. Final remarks
  174. Summary
  175. Questions
  176. Further reading
  177. Conditional Testing and Scripting Loops
  178. Technical requirements
  179. Advanced if-then-else
  180. A recap on if-then-else 
  181. Using regular expressions in tests
  182. The elif condition
  183. Nesting
  184. Getting help
  185. The while loop
  186. The until loop
  187. Creating an interactive while loop
  188. The for loop
  189. Globbing and the for loop
  190. Loop control
  191. Breaking the loop
  192. The continue keyword
  193. Loop control and nesting
  194. Summary
  195. Questions
  196. Further reading
  197. Using Pipes and Redirection in Scripts
  198. Technical requirements
  199. Input/output redirection
  200. File descriptors
  201. Redirecting output
  202. stdout
  203. stderr
  204. Redirect all output
  205. Special output redirection
  206. /dev/null
  207. /dev/zero
  208. Input redirection
  209. Generating a password
  210. Advanced redirecting
  211. Redirecting redirections
  212. Command substitution
  213. Process substitution
  214. Pipes
  215. Binding stdout to stdin
  216. Practical examples
  217. Yet another password generator
  218. Setting passwords in a script
  219. tee
  220. Here documents
  221. Heredocs and variables
  222. Using heredocs for script input
  223. Here strings
  224. Summary
  225. Questions
  226. Further reading
  227. Functions
  228. Technical requirements
  229. Functions explained
  230. Hello world!
  231. More complexity
  232. Variable scopes
  233. Practical examples
  234. Error handling
  235. Augmenting functions with parameters
  236. Colorful
  237. Returning values
  238. Function libraries
  239. Source
  240. More practical examples
  241. Current working directory
  242. Type checking
  243. Yes-no check
  244. Summary
  245. Questions
  246. Further reading
  247. Scheduling and Logging
  248. Technical requirements
  249. Scheduling with at and cron
  250. at
  251. Time syntax
  252. The at queue
  253. at output
  254. cron
  255. crontab
  256. Syntax for the crontab
  257. Logging script results
  258. Crontab environment variables
  259. PATH
  260. SHELL
  261. MAILTO
  262. Logging with redirection
  263. Final logging considerations
  264. A note on verbosity
  265. Summary
  266. Questions
  267. Further reading
  268. Parsing Bash Script Arguments with getopts
  269. Technical requirements
  270. Positional parameters versus flags
  271. Using flags on the command line
  272. The getopts shell builtin
  273. The getopts syntax
  274. Multiple flags
  275. Flags with arguments
  276. Combining flags with positional arguments
  277. Summary
  278. Questions
  279. Further reading
  280. Bash Parameter Substitution and Expansion
  281. Technical requirements
  282. Parameter expansion
  283. Parameter substitutions – recap
  284. Default values
  285. Input checking
  286. Parameter length
  287. Variable manipulation
  288. Pattern substitution
  289. Pattern removal
  290. Case modification
  291. Substring expansion
  292. Summary
  293. Questions
  294. Further reading
  295. Tips and Tricks with Cheat Sheet
  296. Technical requirements
  297. General tips and tricks
  298. Arrays
  299. The history command
  300. Creating your own aliases
  301. Command-line shortcuts
  302. Fun with exclamation marks
  303. Running commands from the history
  304. Keyboard shortcuts
  305. Copying and pasting from the terminal
  306. Reverse search
  307. Cheat sheet for interactive commands
  308. Navigation
  309. cd
  310. ls
  311. pwd
  312. File manipulation
  313. cat
  314. less
  315. touch
  316. mkdir
  317. cp
  318. rm
  319. mv
  320. ln
  321. head
  322. tail
  323. Permissions and ownership
  324. chmod
  325. umask
  326. chown
  327. chgrp
  328. sudo
  329. su
  330. useradd
  331. groupadd
  332. usermod
  333. Summary
  334. Final words
  335. Assessments
  336. Chapter 2
  337. Chapter 3
  338. Chapter 4
  339. Chapter 5
  340. Chapter 6
  341. Chapter 7
  342. Chapter 8
  343. Chapter 9
  344. Chapter 10
  345. Chapter 11
  346. Chapter 12
  347. Chapter 13
  348. Chapter 14
  349. Chapter 15
  350. Chapter 16
  351. Other Books You May Enjoy
  352. Leave a review - let other readers know what you think

chmod, umask

Let's circle back to our testfile. It has the following permissions: -rw-rw----. Read/writable by user and group, readable by others. While these permissions might be fine for most files, they are definitely not a great fit for all files. What about private files? You would not want those to be readable by everyone, perhaps not even by group members.

The Linux command to change permissions on a file or directory is chmod, which we like to read as change file mode. chmod has two operating modes: symbolic mode and numeric/octal mode. We will begin by explaining symbolic mode (which is easier to understand), before we move to octal mode (which is faster to use).

Something we have not yet introduced is the command to view manuals for commands. The command is simply man, followed by the command for which you'd like to see the manual of. In this case, man chmod will place us into the chmod manual pager, which uses the same navigation controls as you learned for Vim. Remember, quitting is done by entering :q. In this case, just q is enough. Take a look at the chmod manual now and read at least the description header; it will make the explanation that follows clearer.

Symbolic mode uses the RWX construct we saw before with the UGOA letters. This might seem new, but it actually isn't! Users, Groups, Others, and All are used to denote which permissions we're changing.

To add permissions, we tell chmod who (users, groups, others, or all) we are doing this for, followed by the permission we want to add. chmod u+x <filename>, for example, will add the execute permission for the user. Similarly, removing permissions with chmod is done as follows: chmod g-rwx <filename>. Notice that we use the + sign to add permissions and the - sign to remove permissions. If we do not specify user, group, others, or all, all is used by default. Let's try this out on our Ubuntu machine:

reader@ubuntu:~$ cd
reader@ubuntu:~$ pwd
/home/reader
reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
-rw-rw-r-- 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$ chmod u+x testfile
reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
-rwxrw-r-- 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$ chmod g-rwx testfile
reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
-rwx---r-- 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$ chmod -r testfile
reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
--wx------ 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$

First, we added the execute permission for the user to the testfile. Next, we removed read, write, and execute from the group, resulting in -rwx---r--. In this scenario, group members are still able to read the file, however, because everyone can still read the file. Not the perfect permissions for privacy, to say the least. Lastly, we do not specify anything before the -r, which effectively removes read access for the user, group, and others, causing the file to end up as --wx------.

Being able to write and execute a file you can't read is a bit weird. Let's fix it and look at how octal permissions work! We can use the verbose option on chmod to make it print more information by using the -v flag:

reader@ubuntu:~$ chmod -v u+rwx testfile
mode of 'testfile' changed from 0300 (-wx------) to 0700 (rwx------)
reader@ubuntu:~$

As you can see, we now get output from chmod! Specifically, we can see the octal mode. Before we changed the file, the mode was 0300, and after adding read for the user, it jumped up to 0700. What do these numbers mean?

It all has to do with the binary implementation of the permission. For all three levels (user, group, others), there are 8 different possible permissions when combining read, write, and execute, as follows:

Symbolic Octal
--- 0
--x 1
-w- 2
-wx 3
r-- 4
r-x 5
rw- 6
rwx 7


Basically, the octal value is between 0 and 7, for a total of 8 values. This is the reason it's called octal: from the Latin/Greek representation of 8, octo. The read permission is given the value of 4, write permission the value of 2, and the execute permission the value of 1.

By using this system, the value of 0 to 7 can always be uniquely related to an RWX value. RWX is 4+2+1 = 7, RX is 4+1 = 5, and so on.

Now that we know how octal representations work, we can use them to modify the file permissions with chmod. Let's give the test file full permissions (RWX or 7) for user, group, and others in a single command:

reader@ubuntu:~$ chmod -v 0777 testfile 
mode of 'testfile' changed from 0700 (rwx------) to 0777 (rwxrwxrwx)
reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
-rwxrwxrwx 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$

In this case, chmod accepts four numbers as the argument. The first number is in regards to a special type of permission called the sticky bit; we won't be discussing this, but we have included material in the Further reading section for those interested. In these examples, it is always set to 0, so no special bits are set. The second number maps to the user permissions, the third to group permissions, and the fourth, unsurprisingly, to the others permissions.

If we wanted to do this using symbolic representation, we could have used the chmod a+rwx command. So, why is octal faster than, as we said earlier on? Let's see what happens if we want to have different permissions for each level, for example, -rwxr-xr--. If we want to do this with symbolic representation, we'd need to use either three commands or one chained command (another function of chmod):

reader@ubuntu:~$ chmod 0000 testfile 
reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
---------- 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$ chmod u+rwx,g+rx,o+r testfile
reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
-rwxr-xr-- 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$

As you can see from the chmod u+rwx,g+rx,o+r testfile command, things have gotten a bit complicated. Using octal notation, however, the command is much simpler:

reader@ubuntu:~$ chmod 0000 testfile 
reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
---------- 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$ chmod 0754 testfile
reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
-rwxr-xr-- 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$

Basically, the difference is mainly using imperative notation (add or remove permissions) versus declarative notation (set it to these values). In our experience, declarative is almost always the better/safer option. With imperative, we need to first check the current permissions and mutate them; with declarative, we can just specify in a single command exactly what we want.

It might be obvious by now, but we prefer to use the octal notation. Besides the benefits from shorter, simpler commands that are handled declaratively, another benefit is that most examples you will find online use the octal notation as well. To fully understand these examples, you will need to at least understand octals. And, if you need to understand them anyway, nothing beats using them in your day to day life!

Earlier, when we used the touch command, we ended up with a file that could be read and written to by both the user and group, and was readable to others. These seem to be default permissions, but where do they come from? And how can we manipulate them? Let's meet umask:

reader@ubuntu:~$ umask
0002
reader@ubuntu:~$

The umask session is used to determine the file permissions for newly created files and directories. For files, the following is done: take the maximum octal value for files, 0666, and subtract the umask (in this case, 0002), which gives us 0664. This would mean that newly created files are -rw-rwr--, which is exactly what we saw for our testfile. Why do we take 0666 and not 0777, you might ask? This is a protection that Linux provides; if we were to use 0777, most files would be created as executable. Executable files can be dangerous, and the design decision was made that files should only be executable when explicitly set that way. So, with the current implementation, there is no such thing as accidentally creating an executable file. For directories, the normal octal value of 0777 is used, which means that directories are created with 0775, -rwxrwxr-x permissions. We can check this out by creating a new directory with the mkdir command:

reader@ubuntu:~$ ls -l
total 4
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
-rwxr-xr-- 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$ umask
0002
reader@ubuntu:~$ mkdir testdir
reader@ubuntu:~$ ls -l
total 8
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
drwxrwxr-x 2 reader reader 4096 Aug 4 16:16 testdir
-rwxr-xr-- 1 reader reader 0 Aug 4 13:44 testfile
reader@ubuntu:~$

Because the execute permission on a directory is much less dangerous (remember, it is used to determine if you can move into the directory), this implementation differs from files.

We have one last trick we'd like to showcase with regards to umask. In specific cases, we'd like to determine default values for files and directories ourselves. We can also do this using the umask command:

reader@ubuntu:~$ umask
0002
reader@ubuntu:~$ umask 0007
reader@ubuntu:~$ umask
0007
reader@ubuntu:~$ touch umaskfile
reader@ubuntu:~$ mkdir umaskdir
reader@ubuntu:~$ ls -l
total 12
-rw-rw-r-- 1 reader reader 69 Jul 14 13:18 nanofile.txt
drwxrwxr-x 2 reader reader 4096 Aug 4 16:16 testdir
-rwxr-xr-- 1 reader reader 0 Aug 4 13:44 testfile
drwxrwx--- 2 reader reader 4096 Aug 4 16:18 umaskdir
-rw-rw---- 1 reader reader 0 Aug 4 16:18 umaskfile
reader@ubuntu:~$

In the preceding example, you can see that running the umask command without arguments prints the current umask. Running it with a valid umask value as an argument changes umask to that value, which is then used when creating new files and directories. Compare umaskfile and umaskdir with the earlier testfile and testdir in the preceding output. This is very useful if we want to create files that are private by default!