Dr Tom Shinder's ISA Server and Beyond : Real World Security Solutions for Microsoft Enterprise Networks

866 Pages·2003·15.342 MB·English

by  Syngress

226_ISA_beyond_FM.qxd 11/8/02 10:10 AM Page i
s o l u t i o n s @ s y n g r e s s . c o m
With more than 1,500,000 copies of our MCSE, MCSD, CompTIA, and Cisco
study guides in print, we continue to look for ways we can better serve the
information needs of our readers. One way we do that is by listening.
Readers like yourself have been telling us they want an Internet-based ser-
vice that would extend and enhance the value of our books. Based on
reader feedback and our own strategic plan, we have created a Web site
that we hope will exceed your expectations.
[email protected] is an interactive treasure trove of useful infor-
mation focusing on our book topics and related technologies. The site
offers the following features:
(cid:2) One-year warranty against content obsolescence due to vendor
product upgrades. You can access online updates for any affected
chapters.
(cid:2) “Ask the Author” customer query forms that enable you to post
questions to our authors and editors.
(cid:2) Exclusive monthly mailings in which our experts provide answers to
reader queries and clear explanations of complex material.
(cid:2) Regularly updated links to sites specially selected by our editors for
readers desiring additional reliable information on key topics.
Best of all, the book you’re now holding is your key to this amazing site.
Just go to www.syngress.com/solutions, and keep this book handy when
you register to verify your purchase.
Thank you for giving us the opportunity to serve your needs. And be sure
to let us know if there’s anything else we can do to help you get the
maximum value from your investment. We’re listening.
www.syngress.com/solutions
226_ISA_beyond_FM.qxd 11/8/02 10:10 AM Page ii
Ultimate CD-ROM Collections Now Available!
Incredible Value for Only $99.99 each
Receive the CD of your choice with seven books in PDF format.
You would pay over $350 for these books in hard copy format.
Ultimate Cisco CD
Managing Cisco Network Security (List Price: $59.95)
Building Cisco Networks for Windows 2000 (List Price: $59.95)
Configuring Cisco AVVID (List Price: $59.95)
Administering Cisco QoS for IP Networks (List Price: $59.95)
Building Cisco Remote Access Networks (List Price: $49.95)
Configuring Cisco Voice Over IP (List Price: $59.95)
Cisco AVVID and IP Telephony Design & Implementation (List Price: $69.95)
Ultimate Windows 2000 CD
Managing Active Directory for Windows 2000 Server (List Price: $49.95)
Managing Windows 2000 Network Services (List Price: $49.95)
Configuring Windows 2000 Server Security (List Price: $49.95)
Windows 2000 Server System Administration Handbook (List Price: $49.95)
Deploying Windows 2000 with Support Tools (List Price: $49.95)
Windows 2000 Configuration Wizards (List Price: $39.95)
Troubleshooting Windows 2000 TCP/IP (List Price: $49.95)
Ultimate .NET CD
BizTalk Server 2000 Developer’s Guide for .NET (List Price: $49.95)
XML .NET Developer’s Guide (List Price: $49.95)
VB .NET Developer’s Guide (List Price: $49.95)
ASP .NET Web Developer’s Guide (List Price: $49.95)
C# .NET Web Developer’s Guide (List Price: $49.95)
.NET Mobile Web Developer’s Guide(List Price: $49.95)
Designing SQL Server 2000 Databases for .NET Enterprise Servers (List Price: $49.95)
Ultimate Network Security CD
Hack Proofing Your Network (List Price: $49.95)
Hack Proofing Windows 2000 Server (List Price: $49.95)
Hack Proofing Linux: A Guide to Open Source Security (List Price: $49.95)
Hack Proofing Sun Solaris (List Price: $49.95)
Hack Proofing Your E-Commerce Site (List Price: $49.95)
Hack Proofing Your Web Applications (List Price $49.95)
Managing Cisco Network Security: Building Rock Solid Networks (List Price $59.95)
VViissiitt wwwwww..ssyynnggrreessss..ccoomm ffoorr ddeettaaiillss..
226_ISA_beyond_FM.qxd 11/8/02 10:10 AM Page iii
1 YEAR UPGRADE
BUYER PROTECTION PLAN
Dr. Tom Shinder’s
ISA Server
Beyond
and
Real World Security Solutions for
Microsoft Enterprise Networks
Thomas W. Shinder, M.D.
Debra Littlejohn Shinder
Martin Grasdal
Technical Editor
226_ISA_beyond_FM.qxd 11/8/02 10:10 AM Page iv
Syngress Publishing,Inc.,the author(s),and any person or firm involved in the writing,editing,or
production (collectively “Makers”) of this book (“the Work”) do not guarantee or warrant the results
to be obtained from the Work.
There is no guarantee of any kind,expressed or implied,regarding the Work or its contents.The Work
is sold AS IS and WITHOUT WARRANTY.You may have other legal rights,which vary from state
to state.
In no event will Makers be liable to you for damages,including any loss of profits,lost savings,or
other incidental or consequential damages arising out from the Work or its contents.Because some
states do not allow the exclusion or limitation of liability for consequential or incidental damages,the
above limitation may not apply to you.
You should always use reasonable care,including backup and other appropriate precautions,when
working with computers,networks,data,and files.
Syngress Media®,Syngress®,“Career Advancement Through Skill Enhancement®,”and “Ask the
Author UPDATE®,”are registered trademarks of Syngress Publishing,Inc.“Mission Critical™,”“Hack
Proofing®,”and “The Only Way to Stop a Hacker is to Think Like One™”are trademarks of Syngress
Publishing,Inc.Brands and product names mentioned in this book are trademarks or service marks of
their respective companies.
KEY SERIAL NUMBER
001 KQMCTFTRCA
002 522CSMBG22
003 A2CY67NGP9
004 NR57JKGLD8
005 T3GF6NV67S
006 XDE4GHK7Z2
007 5T3BY7P88S
008 W4CFD65T9G
009 TFV89V6X5S
010 HUN45F8AS4
PUBLISHED BY
Syngress Publishing,Inc.
800 Hingham Street
Rockland,MA 02370
Dr. Tom Shinder’s ISA Server and Beyond: Real World Security Solutions for Microsoft Enterprise Networks
Copyright © 2002 by Syngress Publishing,Inc.All rights reserved.Printed in the United States of
America.Except as permitted under the Copyright Act of 1976,no part of this publication may be
reproduced or distributed in any form or by any means,or stored in a database or retrieval system,
without the prior written permission of the publisher,with the exception that the program listings
may be entered,stored,and executed in a computer system,but they may not be reproduced for
publication.
Printed in the United States of America
1 2 3 4 5 6 7 8 9 0
ISBN: 1-931836-66-3
Technical Editor:Martin Grasdal Cover Designer:Michael Kavish
Acquisitions Editor:Andrew Williams Page Layout and Art by:Shannon Tozier
Developmental Editor:Jonathan Babcock Copy Editor:Mary Millhollon,Beth Robers
CD Production:Michael Donovan Indexer:Rich Carlson
Distributed by Publishers Group West in the United States and Jaguar Book Group in Canada.
226_ISA_beyond_FM.qxd 11/8/02 10:10 AM Page v
Authors
Thomas W.Shinder,M.D.(MCSE) is a computing industry veteran who
has worked as a trainer,writer,and a consultant for Fortune 500 companies
including FINA Oil,Lucent Technologies,and Sealand Container
Corporation.Tom was Series Editor for the Syngress/Osborne series of
Windows 2000 certification study guides and is author of the best selling book
Configuring ISA Server 2000:Building Firewalls for Windows 2000 (Syngress
Publishing,ISBN:1-928994-29-6).Tom is the editor of the Brainbuzz.com
Win2k News newsletter and is a regular contributor to TechProGuild.He is also
a content editor,contributor,and moderator for the World’s leading site on ISA
Server 2000,www.isaserver.org.Microsoft recognized Tom’s leadership in the
ISA Server community and awarded him their Most Valued Professional
(MVP) award for the first time in December of 2001.
Debra Littlejohn Shinder (MCSE) is author of Scene of the Cybercrime:
Computer Forensics Handbook (Syngress Publishing,ISBN:1-931836-65-5),
co-author of Configuring ISA Server 2000:Building Firewalls for Windows 2000
(Syngress Publishing,ISBN:1-928994-29-6) andTroubleshooting Windows
2000 TCP/IP (Syngress Publishing,ISBN:1-928994-11-3),as well as con-
tributor to numerous other technical books.Along with her husband,Dr.
Thomas W.Shinder,Deb does network consulting in the Dallas-Ft.Worth
area,designs Web sites for businesses,municipalities and non-profit organiza-
tions,and teaches in the Dallas County Community College District’s tech-
nical training programs.As a former police officer and Police Academy
instructor,she specializes in computer/network security and forensics.
Deb has written hundreds of articles for Web and print publications such
as TechRepublic,CNET,Swynk.com,BrainBuzz.com,and WinXP News.She
has also written numerous online courses for DigitalThink,Inc.and prepared
curricula for classroom instruction.She has contributed to Microsoft’s
TechNet,and speaks at conferences such as the BlackHat security briefings
and Certification Expo.She edits the A+ weekly newsletter for CramSession
and writes a weekly feature for the Net Admin News.
Deb has been writing since she finished her first (still unpublished) novel
in ninth grade.She edited her high school and college newspapers and wrote
and edited newsletters for city employees and police associations.Prior to
vv
226_ISA_beyond_FM.qxd 11/8/02 10:10 AM Page vi
entering the tech field,she had articles published in law enforcement and
self-help psychology publications.She is a member of the IEEE’s IPv6
Working Group and has written and tech edited questions for various certifi-
cation practice exams.
Contributor
Mark Burnett is an independent security consultant and freelance writer
who specializes in securing IIS.He is co-author of Maximum Windows
Security and Special OPS:Host and Network Security for Microsoft,UNIX,and
Oracle (Syngress Publishing,ISBN:1-931836-69-8).Mark is a regular con-
tributor to many security-related magazines,newsletters,and Web publica-
tions.As editor of www.iissecurity.net,Mark shares his own unique research
as well as that from security researchers around the globe.
Technical Editor and Contributor
Martin Grasdal (MCSE+I,MCSE/W2K,MCT,CISSP,CTT,A+),Director
of Web Sites and CTO at Brainbuzz.com,has worked in the computer
industry for over nine years.He has been an MCT since 1995 and an MCSE
since 1996.His training and networking experience covers a broad range of
products,including NetWare,Lotus Notes,Windows NT and 2000,
Exchange Server,IIS,Proxy Server,and ISA Server.Martin also works
actively as a consultant.His recent consulting experience includes contract
work for Microsoft as a Technical Contributor to the MCP Program on pro-
jects related to server technologies.Martin has served as Technical Editor for
several Syngress books,including Configuring ISA Server 2000:Building
Firewalls for Windows 2000 (ISBN:1-928994-29-6),and Configuring and
Troubleshooting Windows XP Professional (ISBN:1-928994-80-6).Martin lives
in Edmonton,Alberta,Canada with his wife,Cathy,and their two sons.
vi
226_ISA_beyond_TOC.qxd 11/8/02 9:09 AM Page vii
Contents
About the CD-ROM xiv
Foreword and Acknowledgements xv
Chapter 1 Defending the Network with ISA Server—and Beyond 1
Introduction 2
ISA Server Overview 3
The Increasing Importance of Security 3
ISA Server Authentication 11
Installing ISA Server 15
Planning and Design Issues 15
Installing ISA Server in a Stand-Alone Configuration 30
Upgrading a Stand-Alone ISA Server to an Array Member 37
Installing ISA Server on a Domain Controller 41
Selecting the ISA Server Client 42
Getting Started with ISA Server 46
Installing Service Pack 1 51
What’s Included in SP1 52
Determining the Version of SP1 Installed 53
Downloading and Installing the Service Pack on the ISA Server 54
Upgrading Client Computers to SP1 56
Service Pack 1 Issues 56
Supporting Network Design 57
Using Application Center 2000 with ISA Server 58
ISA Server and Windows .NET 58
Third-Party Software Add-Ons for ISA Server 59
ISA Server Certification 60
Beyond ISA Server 61
Summary,Defensive Tactics Fast Track,Frequently Asked Questions 63
Chapter 2 Defense Plan 1: The Trihomed DMZ 69
Introduction 70
Configuring a Trihomed DMZ 70
The Network Layout 72
ISA 73
Router 74
Configuring the ISA Server 75
Ping Testing the Connections 77
Publishing DMZ SMTP Servers 87
Publishing a DMZ SMTP Mail Relay Server 90
Publishing a Web Server 99
Publishing an FTP Server on a Trihomed DMZ Segment 99
vii
226_ISA_beyond_TOC.qxd 11/8/02 9:09 AM Page viii
viii Contents
How FTP Works 100
Using Packet Filters to Publish the PORT Mode FTP Server 105
Using Packet Filters to Publish the PASV Mode FTP Server 107
External Network Clients Cannot Use the DMZ Interface
to Connect to the Internal Network 110
Summary,Defensive Tactics Fast Track,Frequently Asked Questions 112
Chapter 3 Defense Plan 2: The Back-to-Back DMZ 117
Introduction 118
Configuring a Private Address Back-to-Back DMZ 118
Allowing Inbound VPN Connections
through a Back-to-Back ISA Server DMZ 173
Configuring a Public Address Back-to-Back DMZ 192
Outbound Access for Internal Network
Clients through a Public Address DMZ 195
Inbound VPN Access through the Public Address DMZ 201
Publishing Servers on the Private
Address DMZ on the External ISA Server 203
Summary,Defensive Tactics Fast Track,Frequently Asked Questions 205
Chapter 4 Defense Plan 3: The Internal “Pseudo” DMZ 211
Introduction 212
Using TCP/IP Filtering (TCP/IP Security) 214
Configuring TCP/IP Filtering 215
Summary of TCP/IP Filtering 219
Using Routing and Remote Access Service (RRAS) Packet Filters 219
Configuring RRAS Packet Filters 221
Using IPSec Policies 235
Elements of an IPSec Policy 235
Creating IPSec Policies to Support Communications
between the LATDMZ and the Internal Network 240
Summary,Defensive Tactics Fast Track,Frequently Asked Questions 250
Chapter 5 Defense Plan 4: Advanced Server Publishing 255
Introduction 256
Disabling Socket Pooling 260
Disabling Web and FTP Service Socket Pooling 262
Disabling SMTP and NNTP Service Socket Pooling 263
Disabling IIS Services on the ISA Server 264
Server Publishing 265
Publishing Terminal Services on the Internal Network 266
Publishing Terminal Services on the ISA Server 271
Publishing Terminal Services on Both the ISA Server and
Internal Network 274
Publishing TSAC Sites 275
Publishing FTP Servers on the Internal Network 285
Publishing FTP Servers Co-Located on the ISA Server 295
Using Web Publishing Rules to Allow Secure FTP Access 304
Publishing HTTP and HTTPS (SSL)
226_ISA_beyond_TOC.qxd 11/8/02 9:09 AM Page ix
Contents ix
Servers with Server Publishing Rules 309
Publishing pcAnywhere on the Internal Network 313
Web Publishing 316
Incoming Web Request Listeners 317
Destination Sets 317
Public DNS Entries 318
Private DNS Entries 319
Terminating an SSL Connection at the ISA Server 321
Bridging SSL Connections 337
Secure FTP Connections Using SSL 347
Publishing a Certificate Server 349
Summary,Defensive Tactics Fast Track,Frequently Asked Questions 352
Chapter 6 Defense Plan 5: Protecting Mail Services 357
Introduction 358
Configuring Mail Services on the ISA Server Computer 359
Publishing the IIS SMTP Service on the ISA Server 360
Message Screener on the ISA Server 367
Publishing Exchange Server on the ISA Server 371
Publishing Outlook Web Access on the ISA Server 401
Message Screener on the ISA Server and Exchange Server 411
Configuring Mail Services on the Internal Network 417
Publishing Exchange Server on the Internal Network 417
Exchange RPC Publishing 420
Publishing Outlook Web Access on
the Internal Network Exchange Server 428
Message Screener on the Internal Network Exchange Server 429
GFI’s Mail Security and Mail Essentials for SMTP Servers 433
MailSecurity Versions 434
Installing MailSecurity for SMTP Gateways 434
Configuring MailSecurity 437
Summary,Defensive Tactics Fast Track,Frequently Asked Questions 443
Chapter 7 Understanding Windows Default
Access Control Settings 447
Introduction 448
Overview of Security Groups and Access Control 448
The Administrators Group 450
The Users Group 451
The Power Users Group 452
Configuring Security during Windows 2000 Setup 453
Default File System and Registry Permissions 457
Default User Rights 472
Default Group Membership 479
Summary,Defensive Tactics Fast Track,Frequently Asked Questions 482
Chapter 8 Using the Security Configuration Tool Set 487
Introduction 488
Security Configuration Tool Set 488