Red Hat Linux Security and Optimization

721 Pages·2002·5.094 MB·English

by  Mohammed J. Kabir

4754-2 cover 10/25/01 1:37 PM Page 1
Your Official Red Hat® Linux® Guide to Security and Optimization ® ®
R
eviewed and approved by the experts at Red Hat, this comprehensive
®
guide delivers the know-how you need to improve the performance of
your Red Hat Linux system—and protect it from attacks and break-ins. ®
Red Hat Linux expert Mohammed Kabir starts by showing you how to P R E S S™
tune the kernel and filesystems and optimize network services, from
speeding up Web servers to boosting the performance of Samba. He then aR Mohammed J. Kabir
explains how to secure your Red Hat Linux system, offering hands-on n
e
techniques for network and Internet security as well as in-depth coverage
dd
of Linux firewalls and virtual private networks.
Complete with security utilities and ready-to-run scripts on CD-ROM,
OH
this official Red Hat Linux guide is an indispensable resource.
CD-ROM FEATURES
pa
Security tools, including
Proven Red Hat Linux Performance and Security Solutions
cgichk.pl, gShield, IP tt
(cid:1) Upgrade and configure your hardware to boost performance Filter, John the Ripper, i ®
mL
(cid:1)Customize the kernel and tune the filesystem for optimal results Lids, LSOF, Nessus,
(cid:1) Use JFS and LVM to enhance the filesystem reliability and manageability Netcat, Ngrep, Nmap, i
(cid:1) Tweak Apache, Sendmail, Samba, and NFS servers for increased speed OpenSSH, OpenSSL, in
(cid:1) Protect against root compromises by enabling LIDS and Libsafe in the Postfix, SAINT trial z
u
kernel version, SARA, Snort, a
(cid:1) Use PAM, OpenSSL, shadow passwords, OpenSSH, and xinetd to enhance Swatch, tcpdump, tx
network security Tripwire Open Source i ®
(cid:1) Set up sensible security on Apache and reduce CGI and SSI risks Linux Edition, Vetescan, oS
(cid:1) Secure BIND, Sendmail, ProFTPD, Samba, and NFS servers and Whisker n
e
(cid:1) Create a highly configurable packet filtering firewall to protect your
network Scripts from the book c
(cid:1) Build a secure virtual private network with FreeS/WAN u
Plus a searchable
(cid:1) Use port scanners, password crackers, and CGI scanners to locate vulner- r
e-version of the book
abilities before the hackers do
i
t
MOHAMMED J. KABIR is the founder and CEO of Evoknow, Inc., a company Reader Level y
specializing in customer relationship management software development. IntAerdmveadnicaetde to (cid:17) T Y T O OLS ON
His books include Red Hat Linux 7 Server, Red Hat Linux Administrator’s I C
Handbook, Red Hat Linux Survival Guide, and Apache Server 2 Bible. Shelving Category UR D-
C R
Networking E O
Reviewed and Approved by the Experts at Red Hat S M
$49.99 USA
$74.99 Canada SECURITY TOOLS Linux Solutions from the Experts at Red Hat
ISBN 0-7645-4754-2 £39.99 UK incl.VAT ON CD-ROM
54999
Cover design by
Michael J. Freeland
www.redhat.com Kabir
Cover photo © ®
www.hungryminds.com P R E S S™
H. Armstrong Roberts ®
9 780764 547546 7 85555 04474 6
014754-2 FM.F 11/5/01 9:03 AM Page i
Red Hat Linux
Security and
Optimization
Mohammed J. Kabir
Hungry Minds, Inc.
New York, NY (cid:2)Indianapolis, IN (cid:2)Cleveland, OH
014754-2 FM.F 11/5/01 9:03 AM Page ii
Red HatLinuxSecurity and Optimization Philippines; by Contemporanea de Ediciones for
Published by Venezuela; by Express Computer Distributors for the
Hungry Minds, Inc. Caribbean and West Indies; by Micronesia Media
909 Third Avenue Distributor, Inc. for Micronesia; by Chips
New York, NY 10022 Computadoras S.A. de C.V. for Mexico; by Editorial
Norma de Panama S.A. for Panama; by American
www.hungryminds.com
Bookshops for Finland.
Copyright © 2002 Hungry Minds, Inc. All rights
For general information on Hungry Minds’ products
reserved. No part of this book, including interior
and services please contact our Customer Care
design, cover design, and icons, may be reproduced
department within the U.S. at 800-762-2974, outside
or transmitted in any form, by any means
the U.S. at 317-572-3993 or fax [phone].
(electronic, photocopying, recording, or otherwise)
without the prior written permission of the publisher. For sales inquiries and reseller information,
including discounts, premium and bulk quantity
Library of Congress Control Number: 2001092938
sales, and foreign-language translations, please
ISBN: 0-7645-4754-2 contact our Customer Care department at
Printed in the United States of America 800-434-3422, fax [phone] or write to Hungry
10 9 8 7 6 5 4 3 2 1 Minds, Inc., Attn: Customer Care Department, 10475
1B/SX/RR/QR/IN Crosspoint Boulevard, Indianapolis, IN 46256.
Distributed in the United States by Hungry Minds, For information on licensing foreign or domestic
Inc. rights, please contact our Sub-Rights Customer Care
department at 212-884-5000.
Distributed by CDG Books Canada Inc. for Canada;
by Transworld Publishers Limited in the United For information on using Hungry Minds’ products
Kingdom; by IDG Norge Books for Norway; by IDG and services in the classroom or for ordering
Sweden Books for Sweden; by IDG Books Australia examination copies, please contact our Educational
Publishing Corporation Pty. Ltd. for Australia and Sales department at 800-434-2086 or fax
New Zealand; by TransQuest Publishers Pte Ltd. for 317-572-4005.
Singapore, Malaysia, Thailand, Indonesia, and Hong For press review copies, author interviews, or other
Kong; by Gotop Information Inc. for Taiwan; by ICG publicity information, please contact our Public
Muse, Inc. for Japan; by Intersoft for South Africa; Relations department at 317-572-3168 or fax
by Eyrolles for France; by International Thomson 317-572-4168.
Publishing for Germany, Austria, and Switzerland; For authorization to photocopy items for corporate,
by Distribuidora Cuspide for Argentina; by LR personal, or educational use, please contact
International for Brazil; by Galileo Libros for Chile; Copyright Clearance Center, 222 Rosewood Drive,
by Ediciones ZETA S.C.R. Ltda. for Peru; by WS Danvers, MA 01923, or fax [phone].
Computer Publishing Corporation, Inc., for the
LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY: THE PUBLISHER AND AUTHOR HAVE USED THEIR
BEST EFFORTS IN PREPARING THIS BOOK. THE PUBLISHER AND AUTHOR MAKE NO REPRESENTATIONS
OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS
BOOK AND SPECIFICALLY DISCLAIM ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. THERE ARE NO WARRANTIES WHICH EXTEND BEYOND THE
DESCRIPTIONS CONTAINED IN THIS PARAGRAPH. NO WARRANTY MAY BE CREATED OR EXTENDED BY
SALES REPRESENTATIVES OR WRITTEN SALES MATERIALS. THE ACCURACY AND COMPLETENESS OF
THE INFORMATION PROVIDED HEREIN AND THE OPINIONS STATED HEREIN ARE NOT GUARANTEED OR
WARRANTED TO PRODUCE ANY PARTICULAR RESULTS, AND THE ADVICE AND STRATEGIES
CONTAINED HEREIN MAY NOT BE SUITABLE FOR EVERY INDIVIDUAL. NEITHER THE PUBLISHER NOR
AUTHOR SHALL BE LIABLE FOR ANY LOSS OF PROFIT OR ANY OTHER COMMERCIAL DAMAGES,
INCLUDING BUT NOT LIMITED TO SPECIAL, INCIDENTAL, CONSEQUENTIAL, OR OTHER DAMAGES.
FULFILLMENT OF EACH COUPON OFFER IS THE SOLE RESPONSIBILITY OF THE OFFEROR.
Trademarks:are trademarks or registered trademarks of Hungry Minds, Inc. All other trademarks are the
property of their respective owners. Hungry Minds, Inc., is not associated with any product or vendor
mentioned in this book.
is a trademark of Hungry Minds, Inc.
014754-2 FM.F 11/5/01 9:03 AM Page iii
Credits
ACQUISITIONS EDITOR PROJECT COORDINATOR
Debra Williams Cauley Maridee Ennis
PROJECT EDITOR GRAPHICS AND PRODUCTION
Pat O’Brien SPECIALISTS
Karl Brandt
TECHNICAL EDITORS Stephanie Jumper
Matthew Hayden Laurie Petrone
Sandra “Sam” Moore BrianTorwelle
ErinZeltner
COPY EDITORS
Barry Childs-Helton QUALITY CONTROL TECHNICIANS
Stephanie Provines Laura Albert
Andy Hollandbeck
EDITORIAL MANAGER Carl Pierce
Kyle Looper
PERMISSIONS EDITOR
RED HAT PRESS LIAISON Carmen Krikorian
Lorien Golaski, Red Hat
Communications Manager MEDIA DEVELOPMENT SPECIALIST
Marisa Pearman
SENIOR VICE PRESIDENT, TECHNICAL
PUBLISHING PROOFREADING AND INDEXING
Richard Swadley TECHBOOKS Production Services
VICE PRESIDENT AND PUBLISHER
Mary Bednarek
About the Author
MohammedKabiristhefounderandCEOofEvoknow,Inc.Hiscompanyspecializes
in open-source solutions and customer relationship management software develop-
ment. When he is not busy managing software projects or writing books, he enjoys
traveling around the world. Kabir studied computer engineering at California State
University, Sacramento. He is also the author of Red Hat Linux Server and Apache
Server Bible.He can be reached at [email protected].
014754-2 FM.F 11/5/01 9:03 AM Page iv
014754-2 FM.F 11/5/01 9:03 AM Page v
This book is dedicated to my wife, who proofs my writing, checks my facts,
and writes my dedications.
014754-2 FM.F 11/5/01 9:03 AM Page vi
Preface
This book is focused on two major aspects of Red Hat Linux system administration:
performance tuning and security. The tuning solutions discussed in this book will
help your Red Hat Linux system to have better performance. At the same time, the
practical security solutions discussed in the second half of the book will allow you
to enhance your system security a great deal. If you are looking for time saving,
practical solutions to performance and security issues, read on!
How This Book is Organized
The book has five parts, plus several appendixes.
Part I: System Performance
This part of the book explains the basics of measuring system performance, cus-
tomizing your Red Hat Linux kernel to tune the operating system, tuning your
hard disks, and journaling your filesystem to increase file system reliability and
robustness.
Part II: Network and Service Performance
This part of the book explains how to tune your important network services,
including Apache Web server, Sendmail and postfix mail servers, and Samba and
NFS file and printer sharing services.
Part III: System Security
This part of the book covers how to secure your system using kernel-based Linux
Intrusion Detection System (LIDS) and Libsafe buffer overflow protection mecha-
nisms. Once you have learned to secure your Red Hat Linux kernel, you can secure
your file system using various tools. After securing the kernel and the file system,
you can secure user access to your system using such tools as Pluggable
Authentication Module (PAM), Open Source Secure Socket Layer (OpenSSL), Secure
Remote Password (SRP), and xinetd.
Part IV: Network Service Security
This part of the book shows how to secure your Apache Web server, BIND DNS
server, Sendmail and postfix SMTP server, POP3 mail server, Wu-FTPD and
ProFTPD FTP servers, and Samba and NFS servers.
vi
014754-2 FM.F 11/5/01 9:03 AM Page vii
Preface vii
Part V: Firewalls
This part of the book shows to create packet filtering firewall using iptables, how to
create virtual private networks, and how to use SSL based tunnels to secure access
to system and services. Finally, you will be introduced to an wide array of security
tools such as security assessment (audit) tools, port scanners, log monitoring and
analysis tools, CGI scanners, password crackers, intrusion detection tools, packet
filter tools, and various other security administration utilities.
Appendixes
These elements include important references for Linux network users, plus an
explanation of the attached CD-ROM.
Conventions of This Book
You don’t have to learn any new conventions to read this book. Just remember the
usual rules:
(cid:3) When you are asked to enter a command, you need press the Enter or the
Return key after you type the command at your command prompt.
(cid:3) A monospacedfont is used to denote configuration or code segment.
(cid:3) Text in italic needs to be replaced with relevant information.
Watch for these icons that occasionally highlight paragraphs.
The Note icon indicates that something needs a bit more explanation.
The Tip icon tells you something that is likely to save you some time and
effort.
014754-2 FM.F 11/5/01 9:03 AM Page viii
viii Red Hat Linux Security and Optimization
The Caution icon makes you aware of a potential danger.
The cross-reference icon tells you that you can find additional information
in another chapter.
Tell Us What You Think of This Book
Both Hungry Minds and I want to know what you think of this book. Give us your
feedback. If you are interested in communicating with me directly, send e-mail
messages to [email protected]. I will do my best to respond promptly.
014754-2 FM.F 11/5/01 9:03 AM Page ix
Acknowledgments
While writing this book, I often needed to consult with many developers whose
tools I covered in this book. I want to specially thank a few such developers who
have generously helped me present some of their great work.
Huagang Xie is the creator and chief developer of the LIDS project. Special
thanks to him for responding to my email queries and also providing me with a
great deal of information on the topic.
Timothy K. Tsai, Navjot Singh, and Arash Baratloo are the three members of the
Libsafe team who greatly helped in presenting the Libsafe information. Very special
thanks to Tim for taking the time to promptly respond to my emails and providing
me with a great deal of information on the topic.
I thank both the Red Hat Press and Hungry Minds teams who made this book a
reality. It is impossible to list everyone involved but I must mention the following
kind individuals.
Debra Williams Cauley provided me with this book opportunity and made sure I
saw it through to the end. Thanks, Debra.
Terri Varveris, the acquisitions editor, took over in Debra’s absence. She made
sure I had all the help needed to get this done. Thanks, Terri.
Pat O’Brien, the project development editor, kept this project going. I don’t know
how I could have done this book without his generous help and suggestions every
step of the way. Thanks, Pat.
Matt Hayden, the technical reviewer, provided numerous technical suggestions,
tips, and tricks — many of which have been incorporated in the book. Thanks, Matt.
Sheila Kabir, my wife, had to put up with many long work hours during the few
months it took to write this book. Thank you, sweetheart.
ix