UNIX and Shell Programming

734 Pages·2013·5.811 MB·English

by  B.M. Harwani

UNIX SHELL
AND
PROGRAMMING
B.M. Harwani
Founder & Owner
Microchip Computer Education (MCE)
Ajmer
(cid:4)
3
Oxford University Press is a department of the University of Oxford.
It furthers the University’s objective of excellence in research, scholarship,
and education by publishing worldwide. Oxford is a registered trade mark of
Oxford University Press in the UK and in certain other countries.
Published in India by
Oxford University Press
YMCA Library Building, 1 Jai Singh Road, New Delhi 110001, India
© Oxford University Press 2013
The moral rights of the author/s have been asserted.
First published in 2013
All rights reserved. No part of this publication may be reproduced, stored in
a retrieval system, or transmitted, in any form or by any means, without the
prior permission in writing of Oxford University Press, or as expressly permitted
by law, by licence, or under terms agreed with the appropriate reprographics
rights organization. Enquiries concerning reproduction outside the scope of the
above should be sent to the Rights Department, Oxford University Press, at the
address above.
You must not circulate this work in any other form
and you must impose this same condition on any acquirer.
ISBN-13: 978-0-19-808216-3
ISBN-10: 0-19-808216-9
Typeset in Times
by Quick Sort (India) Private Limited, Chennai
Printed in India by Raj Kamal Electric Press, Kundli, Haryana
Third-party website addresses mentioned in this book are provided
by Oxford University Press in good faith and for information only.
Oxford University Press disclaims any responsibility for the material contained therein.
Dedicated to my mother,
Nita Harwani
Mom, whatever I am today is
because of the moral values taught by you.
I also pay tribute to the offi cers,
men and women of all ranks,
of the Indian Armed Forces.
I salute these brave, patriotic, and disciplined people
for serving our country.
PPrreeffaaccee
Unix operating system, developed in the 1960s, is regarded as one of the most powerful operating
systems, due to its portability and usage in almost all kinds of environments. It is the result of the
combined efforts of many people—students, professors, researchers, and commercial companies. It is a
multitasking and multi-user operating system that is portable on several hardware platforms and is very
secure. It provides a rich set of tools and utilities that help administrators, programmers, and users, to
a great extent, in executing their tasks. Besides this, Unix offers the fl exibility of controlling individual
jobs executed by a user.
Since its inception, Unix has been evolving constantly and has given rise to various products such
as Linux, Ubuntu, FreeBSD, SunOS, Solaris, SCO, and AIX. In order to understand and learn these
products that are widely in use, it is imperative for users to have a clear understanding of the root, that
is, the actual Unix operating system—its features, management of devices and fi les, implementation of
security, scheduling of CPU, and memory management.
Nowadays, Unix and its by-products are used as servers and in developing mobile applications. Unix
has also served as a model for the development of the Internet, thus shifting the focus of computers
towards the creation of networks.
ABOUT THE BOOK
The book has been designed to cater to students, teachers, professionals, and developers to help them
learn the fundamental concepts of the Unix operating system. It follows a bottom-up approach, that is,
it explains basic commands and gradually moves towards advanced commands. Similarly, it begins with
small and easy scripts and makes the reader acquainted with the fundamental statements, loops, and
conditional statements in a systematic manner. Gradually, it moves on to explain large, complex, and
critical scripts. The book focuses on advanced Unix commands that perform critical functions such as
setting access permissions, changing ownerships of the fi les, sharing fi les among groups, performing
input/output (I/O) redirections, cutting or slicing the fi le vertically, pasting content, comparing fi les, and
printing documents. It explains in detail the manipulation of processes and signals and the role of system
calls. All the major editors in Unix, namely, stream editor (sed), visual editor (vi), and modeless editor
(emacs) are explained in detail.
The book describes Bourne, Korn, and C shell programming and covers all important topics and
commands associated with these shells. It also includes numerous programming scripts for better
understanding of the three types of shells. The later part of the book includes dedicated chapters
on language development tools (Yacc, Lex, and M4), text-formatting tools (troff and nroff), and Unix
networking and administration.
KEY FEATURES
The book is packed with numerous student-friendly features that are described here.
(cid:129) Complete scripts along with their outputs are provided for easy implementation of the concepts learnt.
(cid:129) Each command is explained with its syntax with the help of multiple examples.
Preface vii
(cid:129) Several options of a single command have been provided in a tabular format along with their function,
description, and examples for quick understanding and usage.
(cid:129) Numerous notes are interspersed with the text for providing additional relevant information.
(cid:129) Around 1000 solved examples and over 900 end-chapter exercises (with answers to objective-type
questions) are provided.
(cid:129) Specially designed brain teasers are provided at the end of most chapters for the readers to develop
an analytical approach to problem-solving.
(cid:129) A variety of objective-type questions—state true or false, fi ll in the blanks, and multiple-choice
questions—are provided at the end of every chapter for testing the understanding of the concepts
learnt.
(cid:129) Several review questions and programming exercises are provided for the reader to practise the
commands and scripts explained in the chapters.
ONLINE RESOURCES
The companion website of the book, http://oupinheonline.com/book/harwani-unix-shell-programming/
9780198082163, provides the following additional resources:
For faculty
(cid:129) Chapter-wise PowerPoint Slides
(cid:129) Answers to select programming exercises given in the book
For students
(cid:129) Chapter-wise executable and complete shell scripts and codes for all the programs given in the
book
(cid:129) Mail Organizer—a small project that sends mail to the desired recipient on a given date
(cid:129) Inventory Management System—a small project that explains maintenance of inventory using
MySQL database server
(cid:129) Debugging exercises with solutions
(cid:129) Flashcards—for active recall of all important Unix commands
ORGANIZATION OF THE BOOK
The book is organized into 15 chapters.
Chapter 1, Unix: An Introduction, focuses on the fundamentals of operating systems, history of Unix,
structure of the Unix operating system, Unix environment, and different types of shells.
Chapter 2, Unix File System, explains the different types of regular and device fi les, organization of a
fi le system, accessing, mounting, and unmounting a fi le system, different blocks of a fi le system, and
structure of inode blocks.
Chapter 3, Basic Unix Commands, describes basic commands such as logging into the system, changing
the password, checking who is logged in, displaying date and time of the system, and dealing with
fi le operations such as creating fi les, displaying their contents, deleting fi les, creating links to fi les,
renaming fi les, and moving fi les. The chapter also explains commands for maintaining directories,
creating a directory, changing the current directory, removing a directory, displaying calendars, using
viii Preface
basic calculators, displaying information about current systems, deleting symbolic links, and exiting
from a Unix system.
Chapter 4, Advanced Unix Commands, discusses advanced commands such as setting access permissions
for the existing fi les and directories, setting default permissions for the newly created fi les and directories,
creating groups, changing ownerships of the fi les, and sharing fi les among groups. The chapter covers
commands for sorting content, performing I/O redirections, cutting the fi le vertically, pasting content,
splitting fi les, counting characters, words, and lines in fi les, using the pipe operator, comparing fi les,
eliminating and displaying duplicate lines, among others.
Chapter 5, File Management and Compression Techniques, explains the types of devices, role of device
drivers, and the way in which devices are represented in the Unix operating system. It details different
disk-related commands required for copying, formatting, fi nding usage, fi nding free space, and making
partitions. It also covers compression and decompression of fi les.
Chapter 6, Manipulating Processes and Signals, focuses on processes and their address space, structure,
data structures describing the processes and process states, commands related to scheduling processes
at the desired time, handling jobs, and switching jobs from the foreground to the background and vice
versa. It explains suspending, resuming, and terminating jobs, executing commands in a batch, ensuring
process execution even when a user logs out, increasing and decreasing the priority of processes, and
killing processes. The chapter also discusses signals, their types, and the methods of signal generation,
virtual memory and its role in executing large applications in a limited physical memory, and mapping
of a virtual address to the physical memory.
Chapter 7, System Calls, is devoted to the role of system calls in performing different tasks. The chapter
explains system calls that are used in fi le handling operations such as opening, creating, reading from
and writing to fi les, closing, deleting, and linking to fi les, changing fi le access permissions, accessing
fi le information, and relocating and duplicating fi le descriptors. The chapter covers the system calls that
perform different tasks related to directory handling such as changing, opening, and reading directories.
The chapter throws light on the system calls involved in process handling operations such as the exec(),
fork, and wait system calls and those that deal with memory management—allocating memory, freeing
memory, changing the size of the allocated memory, fi le locking, and record locking .
Chapter 8, Editors in Unix, explains the usage of the stream editor (sed) in fi ltering out the desired data
from the specifi ed fi le, inserting lines, deleting lines, saving fi ltered content into another fi le, loading the
content of another fi le into the current fi le, and searching for content that matches specifi c patterns. The
chapter also explains the visual editor (vi) and the modeless editor (emacs).
Chapter 9, AWK Script, discusses the role of the AWK scripts in fi ltering and processing content. It
explains the different functions used in AWK for printing results, formatting output, and searching for
desired patterns. The chapter also details different operators (comparison, logical, arithmetic), functions
(string, arithmetic, and search and substitute), and built-in variables to perform the desired operations
quickly and with the least effort. It also discusses different loops to perform repetitive tasks, taking input
from the user to perform operations on the desired content.
Chapter 10, Bourne Shell Programming, explains different command line parameters used in Bourne
shell scripts, conditional statements, loops, reading input, displaying output, testing data, translating
content, and searching for patterns in fi les. The chapter also covers displaying the exit status of the
Preface ix
commands, applying command substitution, sending and receiving messages between users, creating
and using functions, setting and displaying terminal confi gurations, managing positional parameters,
and using fetch options in the command line.
Chapter 11, Korn Shell Programming, helps us in understanding different features of the Korn shell,
command line editing, fi le name completion, command name aliasing, command history substitution,
and meta characters. It explains different operators, shell variables, basic I/O commands, command line
arguments, if else and case statements, strings, fi les, loops, arrays, functions, and I/O redirection.
Chapter 12, C Shell Programming, describes the C shell and its different features. The chapter explains
command history, command substitution, fi lename substitution (globbing), fi lename completion, and
aliases. It also covers job control, running jobs in the background, and suspending, resuming, and killing
jobs. It aids in the understanding of environment variables, shell variables, built-in shell variables, and
customizing the shell and C shell operators. The chapter also discusses different fl ow control statements,
loops, arrays, and errors.
Chapter 13, Different Tools and Debuggers, describes language development tools Yacc, Lex, and M4 and
text-formatting tools, troff and nroff. The chapter covers different preprocessors for nroff and troff
such as tbl, eqn, and pic. The chapter also discusses debugger tools, dbx, adb, and sdb.
Chapter 14, Interprocess Communication, covers pipes and messages as also accessing, attaching,
reading, writing, and detaching the shared memory segment. It helps the readers in getting acquainted
with initializing, managing, and performing operations on sockets (stream and datagram), I/O
multiplexing, fi lters, and semaphores.
Chapter 15, Unix System Administration and Networking, discusses the Unix booting procedure,
mounting and unmounting fi le systems, managing user accounts, network security, and backup and
restore.
ACKNOWLEDGEMENTS
I thank my family, my small world: my wife, Anushka and my wonderful children, Chirag and Naman
for inspiring and motivating me and forgiving me for spending long hours on the computer during
the course of development of this book.
Speaking of encouragement, I must thank my students who, with their innumerable queries, helped
me understand the essential expectations of a reader. This in turn made me add numerous examples
and exercises, thus giving a practical approach to the book.
My acknowledgements would remain incomplete if I did not thank the editorial team at Oxford
University Press, India, who supported me throughout the development of this book. My special
thanks are due to the reviewers for their constructive comments and valuable suggestions.
I have tried to cover the necessary topics and explain them in a simple and user-friendly manner.
Any comments or suggestions that can be incorporated in future editions of this book may be sent to me
at [email protected].
B.M. Harwani
BBrriieeff CCoonntteennttss
Features of the Book iv
Preface vi
Detailed Contents xi
1. Unix: An Introduction 1
2. Unix File System 13
3. Basic Unix Commands 27
4. Advanced Unix Commands 59
5. File Management and Compression Techniques 94
6. Manipulating Processes and Signals 148
7. System Calls 192
8. Editors in Unix 258
9. AWK Script 305
10. Bourne Shell Programming 378
11. Korn Shell Programming 480
12. C Shell Programming 558
13. Different Tools and Debuggers 624
14. Interprocess Communication 653
15. Unix System Administration and Networking 672
Index 697
DDeettaaiilleedd CCoonntteennttss
Features of the Book iv
Preface vi
Brief Contents x
1. Unix: An Introduction 1 2.4.3 Inode Block 21
2.4.4 Data Block 24
1.1 Operating System 1
1.1.1 Functions of Operating Systems 2
3. Basic Unix Commands 27
1.2 History of Unix 3
1.3 Overview and Features of 3.1 login: Logging in to Systems 27
Unix System 4 3.2 Overview of Commands 28
1.3.1 Multitasking 4 3.2.1 Structure 29
1.3.2 Multi-user 5 3.2.2 Types of Commands in Unix 29
1.3.3 Portability 5
1.3.4 Job Control 5 4. Advanced Unix Commands 59
1.3.5 Tools and Utilities 5 4.1 Overview 59
1.3.6 Security 6 4.2 File Access Permissions 60
1.4 Structure of Unix System 6 4.2.1 chmod: Changing File
1.4.1 Hardware 6 Access Permissions 61
1.4.2 Kernel 7 4.2.2 umask: Setting Default
1.4.3 Shell 8 Permissions 62
1.4.4 Tools and Applications 9 4.2.3 chown: Changing File Ownership 64
1.5 Unix Environment 9 4.2.4 chgrp: Changing Group
1.5.1 Stand-alone Personal Environment 10 Command 65
1.5.2 Time-sharing Environment 10 4.2.5 groups: Displaying Group
1.5.3 Client–Server Environment 10 Membership 66
4.2.6 groups: Sharing
2. Unix File System 13
Files Among Groups 66
2.1 Introduction to Files 13 4.3 Input/Output Redirection
2.1.1 Types of Files 13 in Unix 67
2.1.2 Symbolic Links 15 4.3.1 Output Redirection Operator 67
2.1.3 Pipes 15 4.3.2 Input Redirection Operator 68
2.1.4 Sockets 16 4.4 Pipe Operator 68
2.2 Organization of File Systems 16 4.5 cut: Cutting Data from Files 68
2.3 Accessing File Systems 17 4.6 paste: Pasting Data in Files 71
2.3.1 Mounting File Systems 18 4.7 split: Splitting Files into
2.3.2 Unmounting File Systems 18 Lines or Bytes 71
2.4 Structure of File Systems 20 4.8 wc: Counting Characters,
2.4.1 Boot Block 20 Words, and Lines in Files 73
2.4.2 Super Block 20 4.9 sort: Sorting Files 73
xii Detailed Contents
4.10 head: Displaying Top 5.4.2 gunzip Command 107
Contents of Files 74 5.4.3 zip Command 109
4.11 tail: Displaying Bottom 5.4.4 unzip Command 111
Contents of Files 75 5.4.5 compress Command 111
4.12 diff: Finding Differences 5.4.6 uncompress Command 114
between Two Files 75 5.4.7 pack Command 115
4.13 cmp: Comparing Files 77 5.4.8 unpack Command 115
4.14 uniq: Eliminating and 5.4.9 bzip2 and bunzip2 Commands 117
Displaying Duplicate Lines 78 5.4.10 bunzip2 Command 119
4.15 comm: Displaying and 5.4.11 7-zip—Implementing
Suppressing Unique or Maximum Compression 119
Common Content in Two Files 79 5.5 Dealing with Files 123
4.16 time: Finding Consumed Time 81 5.5.1 fi le: Determining File Type 124
4.17 pg: Showing Content Page-wise 82 5.5.2 fi nd: Locating Files 124
4.18 lp: Printing Documents 82 5.5.3 locate: Searching for
4.19 cancel: Cancelling Files with Specifi c Strings 129
Print Command 84 5.5.4 which/whence: Finding
4.20 Understanding .profi le Files 84 Locations of Programs or
4.21 calendar: Getting Reminders 85 Utilities on Disks 130
4.22 script: Recording Sessions 85 5.5.5 fsck: Utility for Checking
4.23 Conversions between File Systems 130
DOS and Unix 86 5.6 Important Unix System Files 135
4.24 man: Displaying Manual 87 5.6.1 /etc/passwd 135
4.25 Correcting Typing Mistakes 88 5.6.2 /etc/shadow 136
5.6.3 /etc/hosts 136
5. File Management and
5.6.4 / etc/hosts.allow and
Compression Techniques 94
/etc/hosts.deny 137
5.1 Managing and Compressing Files 94 5.7 Shell Variables 138
5.2 Computer Devices 95
5.7.1 User-created Shell Variables 138
5.2.1 Dealing with Devices 96 5.7.2 System Shell Variables 138
5.2.2 Block device 97 5.8 Export of Local and
5.2.3 Major and Minor Numbers 98 Global Shell Variables 141
5.3 Disk-related Commands 98
6. Manipulating Processes
5.3.1 dd: Copying Disks 99
and Signals 148
5.3.2 du: Disk Usage 99
5.3.3 df: Reporting Free and 6.1 Process Basics 148
Available Space on File Systems 101 6.1.1 Process Address Space 151
5.3.4 dfspace: Reporting Free 6.1.2 Process Structure 151
Space on File Systems 103 6.1.3 Creation and
5.3.5 fdisk: Dividing Termination of Processes 154
Disks into Partitions 103 6.2 Process States and Transitions 154
5.4 Compressing and Uncompressing Files 105 6.3 Zombie Process 156
5.4.1 gzip Command 105 6.4 Context Switching 157