Table of Contents for
sed & awk, 2nd Edition

Version ebook / Retour

Cover image for bash Cookbook, 2nd Edition sed & awk, 2nd Edition by Arnold Robbins Published by O'Reilly Media, Inc., 1997
  1. sed & awk, 2nd Edition
  2. Cover
  3. sed & awk, 2nd Edition
  4. A Note Regarding Supplemental Files
  5. Dedication
  6. Preface
  7. Scope of This Handbook
  8. Availability of sed and awk
  9. Obtaining Example Source Code
  10. Conventions Used in This Handbook
  11. About the Second Edition
  12. Acknowledgments from the First Edition
  13. Comments and Questions
  14. 1. Power Tools for Editing
  15. 1.1. May You Solve Interesting Problems
  16. 1.2. A Stream Editor
  17. 1.3. A Pattern-Matching Programming Language
  18. 1.4. Four Hurdles to Mastering sed and awk
  19. 2. Understanding Basic Operations
  20. 2.1. Awk, by Sed and Grep, out of Ed
  21. 2.2. Command-Line Syntax
  22. 2.3. Using sed
  23. 2.4. Using awk
  24. 2.5. Using sed and awk Together
  25. 3. Understanding Regular Expression Syntax
  26. 3.1. That’s an Expression
  27. 3.2. A Line-Up of Characters
  28. 3.3. I Never Metacharacter I Didn’t Like
  29. 4. Writing sed Scripts
  30. 4.1. Applying Commands in a Script
  31. 4.2. A Global Perspective on Addressing
  32. 4.3. Testing and Saving Output
  33. 4.4. Four Types of sed Scripts
  34. 4.5. Getting to the PromiSed Land
  35. 5. Basic sed Commands
  36. 5.1. About the Syntax of sed Commands
  37. 5.2. Comment
  38. 5.3. Substitution
  39. 5.4. Delete
  40. 5.5. Append, Insert, and Change
  41. 5.6. List
  42. 5.7. Transform
  43. 5.8. Print
  44. 5.9. Print Line Number
  45. 5.10. Next
  46. 5.11. Reading and Writing Files
  47. 5.12. Quit
  48. 6. Advanced sed Commands
  49. 6.1. Multiline Pattern Space
  50. 6.2. A Case for Study
  51. 6.3. Hold That Line
  52. 6.4. Advanced Flow Control Commands
  53. 6.5. To Join a Phrase
  54. 7. Writing Scripts for awk
  55. 7.1. Playing the Game
  56. 7.2. Hello, World
  57. 7.3. Awk’s Programming Model
  58. 7.4. Pattern Matching
  59. 7.5. Records and Fields
  60. 7.6. Expressions
  61. 7.7. System Variables
  62. 7.8. Relational and Boolean Operators
  63. 7.9. Formatted Printing
  64. 7.10. Passing Parameters Into a Script
  65. 7.11. Information Retrieval
  66. 8. Conditionals, Loops, and Arrays
  67. 8.1. Conditional Statements
  68. 8.2. Looping
  69. 8.3. Other Statements That Affect Flow Control
  70. 8.4. Arrays
  71. 8.5. An Acronym Processor
  72. 8.6. System Variables That Are Arrays
  73. 9. Functions
  74. 9.1. Arithmetic Functions
  75. 9.2. String Functions
  76. 9.3. Writing Your Own Functions
  77. 10. The Bottom Drawer
  78. 10.1. The getline Function
  79. 10.2. The close( ) Function
  80. 10.3. The system( ) Function
  81. 10.4. A Menu-Based Command Generator
  82. 10.5. Directing Output to Files and Pipes
  83. 10.6. Generating Columnar Reports
  84. 10.7. Debugging
  85. 10.8. Limitations
  86. 10.9. Invoking awk Using the #! Syntax
  87. 11. A Flock of awks
  88. 11.1. Original awk
  89. 11.2. Freely Available awks
  90. 11.3. Commercial awks
  91. 11.4. Epilogue
  92. 12. Full-Featured Applications
  93. 12.1. An Interactive Spelling Checker
  94. 12.2. Generating a Formatted Index
  95. 12.3. Spare Details of the masterindex Program
  96. 13. A Miscellany of Scripts
  97. 13.1. uutot.awk—Report UUCP Statistics
  98. 13.2. phonebill—Track Phone Usage
  99. 13.3. combine—Extract Multipart uuencoded Binaries
  100. 13.4. mailavg—Check Size of Mailboxes
  101. 13.5. adj—Adjust Lines for Text Files
  102. 13.6. readsource—Format Program Source Files for troff
  103. 13.7. gent—Get a termcap Entry
  104. 13.8. plpr—lpr Preprocessor
  105. 13.9. transpose—Perform a Matrix Transposition
  106. 13.10. m1—Simple Macro Processor
  107. A. Quick Reference for sed
  108. A.1. Command-Line Syntax
  109. A.2. Syntax of sed Commands
  110. A.3. Command Summary for sed
  111. B. Quick Reference for awk
  112. B.1. Command-Line Syntax
  113. B.2. Language Summary for awk
  114. B.3. Command Summary for awk
  115. C. Supplement for Chapter 12
  116. C.1. Full Listing of spellcheck.awk
  117. C.2. Listing of masterindex Shell Script
  118. C.3. Documentation for masterindex
  119. masterindex
  120. C.3.1. Background Details
  121. C.3.2. Coding Index Entries
  122. C.3.3. Output Format
  123. C.3.4. Compiling a Master Index
  124. Index
  125. About the Authors
  126. Colophon
  127. Copyright

uutot.awk—Report UUCP Statistics

Contributed by Roger A. Cornelius

Here’s something I wrote in nawk in response to all the C versions of the same thing which were posted to alt.sources awhile back. Basically, it summarizes statistics of uucp connections (connect time, throughput, files transmitted, etc.). It only supports HDB-style log files, but will show statistics on a site-by-site, or on an overall (all sites), basis. [It also works with /usr/spool/uucp/SYSLOG.]

I use a shell wrapper which calls “awk -f” to run this, but it’s not necessary. Usage information is in the header. (Sorry about the lack of comments.)

# @(#) uutot.awk - display uucp statistics - requires new awk
# @(#) Usage:awk -f uutot.awk [site ...] /usr/spool/uucp/.Admin/xferstats
# Author: Roger A. Cornelius (rac@sherpa.uucp)

#       dosome[];               # site names to work for - all if not set
#       remote[];               # array of site names
#       bytes[];                # bytes xmitted by site
#       time[];	               # time spent by site
#       files[];                # files xmitted by site
BEGIN {
	doall = 1;
	if (ARGC > 2) {
		doall = 0;
		for (i = 1; i < ARGC-1; i++) {
			dosome[ ARGV[i] ];
			ARGV[i] = "";
		}
	}

	kbyte = 1024	# 1000 if you're not picky
	bang = "!";
	sending = "->";
	xmitting = "->" "|" "<-";

	hdr1 = "Remote     K-Bytes   K-Bytes   K-Bytes " \
		"Hr:Mn:Sc Hr:Mn:Sc AvCPS AvCPS    #    #\n";
	hdr2 = "SiteName      Recv      Xmit     Total     " \
		"Recv     Xmit  Recv  Xmit Recv Xmit\n";

	hdr3 = "-------- --------- --------- --------- -------- " \
		"-------- ----- ----- ---- ----";
	fmt1 = "%-8.8s %9.3f %9.3f %9.3f %2d:%02d:%02.0f " \
		"%2d:%02d:%02.0f %5.0f %5.0f %4d %4d\n";
	fmt2 = "Totals   %9.3f %9.3f %9.3f %2d:%02d:%02.0f " \
		"%2d:%02d:%02.0f %5.0f %5.0f %4d %4d\n";
}
{
	if ($6 !~ xmitting)		# should never be
		next;
	direction = ($6 == sending ? 1 : 2)

	site = substr($1,1,index($1,bang)-1);
	if (site in dosome || doall) {
		remote[site];
		bytes[site,direction] += $7;
		time[site,direction] += $9;
		files[site,direction]++;
	}
}
END {
	print hdr1 hdr2 hdr3;
	for (k in remote) {
		rbyte += bytes[k,2];	sbyte += bytes[k,1];
		rtime += time[k,2];	stime += time[k,1];
		rfiles += files[k,2];	sfiles += files[k,1];
		printf(fmt1, k, bytes[k,2]/kbyte, bytes[k,1]/kbyte,
			(bytes[k,2]+bytes[k,1])/kbyte,
			time[k,2]/3600, (time[k,2]%3600)/60, time[k,2]%60,
			time[k,1]/3600, (time[k,1]%3600)/60, time[k,1]%60,
			bytes[k,2] && time[k,2] ? bytes[k,2]/time[k,2] : 0,
			bytes[k,1] && time[k,1] ? bytes[k,1]/time[k,1] : 0,
			files[k,2], files[k,1]);
	}

	print hdr3
	printf(fmt2, rbyte/kbyte, sbyte/kbyte, (rbyte+sbyte)/kbyte,
		rtime/3600, (rtime%3600)/60, rtime%60,
		stime/3600, (stime%3600)/60, stime%60,
		rbyte && rtime ? rbyte/rtime : 0,
		sbyte && stime ? sbyte/stime : 0,
		rfiles, sfiles);
}

A test file was generated to test Cornelius’ program. Here are a few lines extracted from /usr/spool/uucp/.Admin/xferstats (because each line in this file is too long to print on a page, we have broken the line following the directional arrow for display purposes only):

isla!nuucp S (8/3-16:10:17) (C,126,25) [ttyi1j] ->
                     1131/4.880 secs, 231 bytes/sec
isla!nuucp S (8/3-16:10:20) (C,126,26) [ttyi1j] ->
                     149/0.500 secs, 298 bytes/sec

isla!sue S (8/3-16:10:49) (C,126,27) [ttyi1j] ->
                     646/25.230 secs, 25 bytes/sec
isla!sue S (8/3-16:10:52) (C,126,28) [ttyi1j] ->
                     145/0.510 secs, 284 bytes/sec
uunet!uisla M (8/3-16:15:50) (C,951,1) [cui1a] ->
                     1191/0.660 secs, 1804 bytes/sec
uunet!uisla M (8/3-16:15:53) (C,951,2) [cui1a] ->
                     148/0.080 secs, 1850 bytes/sec
uunet!uisla M (8/3-16:15:57) (C,951,3) [cui1a] ->
                     1018/0.550 secs, 1850 bytes/sec
uunet!uisla M (8/3-16:16:00) (C,951,4) [cui1a] ->
                     160/0.070 secs, 2285 bytes/sec
uunet!daemon M (8/3-16:16:06) (C,951,5) [cui1a] <-
                     552/2.740 secs, 201 bytes/sec
uunet!daemon M (8/3-16:16:09) (C,951,6) [cui1a] <-
                     102/1.390 secs, 73 bytes/sec

Note that there are 12 fields; however, the program really only uses fields 1, 6, 7, and 9. Running the program on the sample input produces the following results:

$ nawk -f uutot.awk uutot.test
Remote     K-Bytes   K-Bytes   K-Bytes Hr:Mn:Sc Hr:Mn:Sc AvCPS AvCPS    #    #
SiteName      Recv      Xmit     Total     Recv     Xmit  Recv  Xmit Recv Xmit
-------- --------- --------- --------- -------- -------- ----- ----- ---- ----
uunet        0.639     2.458     3.097  0:04:34  2:09:49     2     0    2    4
isla         0.000     2.022     2.022  0:00:00  0:13:58     0     2    0    4
-------- --------- --------- --------- -------- -------- ----- ----- ---- ----
Totals       0.639     4.480     5.119  0:04:34  2:23:47     2     1    2    8

Program Notes for uutot.awk

This nawk application is an excellent example of a clearly written awk program. It is also a typical example of using awk to change a rather obscure UNIX log into a useful report.

Although Cornelius apologizes for the lack of comments that explain the logic of the program, the usage of the program is clear from the initial comments. Also, he uses variables to define search patterns and the report’s layout. This helps to simplify conditional and print statements in the body of the program. It also helps that the variables have names which aid in immediately recognizing their purpose.

This program has a three-part structure, as we emphasized in Chapter 7. It consists of a BEGIN procedure, in which variables are defined; the body, in which each line of data from the log file is processed; and the END procedure, in which the output for the report is generated.