Solaris performance and tools DTrace and MDB techniques for Solaris 10 and OpenSolaris

490 Pages·2.088 MB·English

by  McDougall, Richard;Mauro, Jim;Gregg, Brendan

Solaris™ Performance
and Tools

Solaris™ Performance
and Tools
DTrace and MDB Techniques for
Solaris 10 and OpenSolaris
Richard McDougall
Jim Mauro
Brendan Gregg
Sun Microsystems Press
Upper Saddle River, NJ • Boston (cid:129) Indianapolis (cid:129) San Francisco
New York (cid:129) Toronto (cid:129) Montreal (cid:129) London (cid:129) Munich (cid:129) Paris (cid:129) Madrid
Capetown (cid:129) Sydney (cid:129) Tokyo (cid:129) Singapore (cid:129) Mexico City
Copyright © 2007 Sun Microsystems, Inc.
4150 Network Circle, Santa Clara, California 95054 U.S.A.
All rights reserved.
Sun Microsystems, Inc., has intellectual property rights relating to implementations of the technology described
in this publication. In particular, and without limitation, these intellectual property rights may include one or
more U.S. patents, foreign patents, or pending applications. Sun, Sun Microsystems, the Sun logo, J2ME, Solaris,
Java, Javadoc, NetBeans, and all Sun and Java based trademarks and logos are trademarks or registered trade-
marks of Sun Microsystems, Inc., in the United States and other countries. UNIX is a registered trademark in the
United States and other countries, exclusively licensed through X/Open Company, Ltd.
THIS PUBLICATION IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. THIS PUBLICATION COULD INCLUDE
TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLY ADDED TO
THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THE
PUBLICATION. SUN MICROSYSTEMS, INC., MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE
PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED IN THIS PUBLICATION AT ANY TIME.
The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales,
which may include electronic versions and/or custom covers and content particular to your business, training
goals, marketing focus, and branding interests. For more information, please contact: U.S. Corporate and Govern-
ment Sales, [phone], [email protected].
For sales outside the U.S., please contact International Sales, [email protected].
Visit us on the Web: www.prenhallprofessional.com
This Book Is Safari Enabled
The Safari® Enabled icon on the cover of your favorite technology book means the book is avail-
able through Safari Bookshelf. When you buy this book, you get free access to the online edition
for 45 days.
Safari Bookshelf is an electronic reference library that lets you easily search thousands of technical books, find
code samples, download chapters, and access technical information whenever and wherever you need it.
To gain 45-day Safari Enabled access to this book:
(cid:129) Go to http://www.prenhallprofessional.com/safarienabled
(cid:129) Complete the brief registration form
(cid:129) Enter the coupon code YXM9-9VPM-QQBK-52MT-GPBD
If you have difficulty registering on Safari Bookshelf or accessing the online edition, please e-mail
[email protected].
Library of Congress Cataloging-in-Publication Data
McDougall, Richard.
Solaris performance and tools : DTrace and MDB techniques for
Solaris 10 and OpenSolaris / Richard McDougall, Jim Mauro,
Brendan Gregg.
p. cm.
Includes bibliographical references and index.
ISBN 0-13-156819-1 (hardback : alk. paper)
1. Solaris (Computer file) 2. Operating systems (Computers)
I. Mauro, Jim. II. Gregg, Brendan. III. Title.
QA76.76.O63M3957 2006
005.4'32—dc22
200602013
All rights reserved. Printed in the United States of America. This publication is protected by copyright, and per-
mission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system,
or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For
information regarding permissions, write to:
Pearson Education, Inc.
Rights and Contracts Department
One Lake Street
Upper Saddle River, NJ 07458
Fax: [phone]
ISBN 0-13-156819-1
Text printed in the United States on recycled paper at Courier in Westford, Massachusetts.
First printing, July 2006
For Traci, Madi, and Boston—
for your love, encouragement, and support . . .
—Richard
Once again . . .
For Donna, Frank, and Dominick.
All my love, always . . .
—Jim

Contents
Foreword xxi
Preface xxiii
About the Authors xxxi
Acknowledgments xxxiii
PART ONE
Observability Methods 1
Chapter 1 Introduction to Observability Tools 3
1.1 Observability Tools 4
1.1.1 Kstat Tools 5
1.1.2 Process Tools 5
1.1.3 Forensic Tools 6
1.2 Drill-Down Analysis 7
1.3 About Part One 8
1.3.1 Chapter Layout 8
Chapter 2 CPUs 11
2.1 Tools for CPU Analysis 11
2.2 vmstat Tool 13
vii
viii Contents
2.3 CPU Utilization 14
2.4 CPU Saturation 15
2.5 psrinfo Command 15
2.6 uptime Command 15
2.7 sar Command 16
2.7.1 sar Default Output 17
2.7.2 sar -q 18
2.7.3 Capturing Historical Data 18
2.8 Clock Tick Woes 19
2.9 mpstat Command 20
2.10 Who Is Using the CPU? 23
2.11 CPU Run Queue Latency 24
2.12 CPU Statistics Internals 26
2.12.1 usr, sys, idl Times 26
2.12.2 Load Averages 28
2.12.3 pr_pctcpu Field 28
2.13 Using DTrace to Explain Events from Performance Tools 29
2.14 DTrace Versions of runq-sz, %runocc 31
2.15 DTrace Probes for CPU States 33
Chapter 3 Processes 35
3.1 Tools for Process Analysis 35
3.2 Process Statistics Summary: prstat 37
3.2.1 Thread Summary: prstat -L 38
3.2.2 Process Microstates: prstat -m 38
3.2.3 Sorting by a Key: prstat -s 39
3.2.4 User Summary: prstat -t 40
3.2.5 Project Summary: prstat -J 40
3.2.6 Zone Summary: prstat -Z 41
3.3 Process Status: ps 41
3.3.1 /usr/bin/ps Command 42
3.3.2 /usr/ucb/ps 44
3.4 Tools for Listing and Controlling Processes 45
3.4.1 Process Tree: ptree 45
3.4.2 Grepping for Processes: pgrep 46
3.4.3 Killing Processes: pkill 46
Contents ix
3.4.4 Temporarily Stop a Process: pstop 47
3.4.5 Making a Process Runnable: prun 47
3.4.6 Wait for Process Completion: pwait 47
3.4.7 Reap a Zombie Process: preap 47
3.5 Process Introspection Commands 47
3.5.1 Process Stack: pstack 48
3.5.2 Process Memory Map: pmap -x 48
3.5.3 Process File Table: pfiles 49
3.5.4 Execution Time Statistics for a Process: ptime 50
3.5.5 Process Signal Disposition: psig 50
3.5.6 Process Libraries: pldd 51
3.5.7 Process Flags: pflags 51
3.5.8 Process Credentials: pcred 51
3.5.9 Process Arguments: pargs 52
3.5.10 Process Working Directory: pwdx 52
3.6 Examining User-Level Locks in a Process 52
3.7 Tracing Processes 53
3.7.1 Using truss to Trace Processes 54
3.7.2 Using apptrace to Trace Processes 55
3.7.3 Using DTrace to Trace Process Functions 57
3.7.4 Using DTrace to Aggregate Process Functions 58
3.7.5 Using DTrace to Peer Inside Processes 58
3.7.6 Using DTrace to Sample Stack Backtraces 59
3.8 Java Processes 60
3.8.1 Process Stack on a Java Virtual Machine: pstack 60
3.8.2 JVM Profiling 61
3.8.3 Tuning Java Garbage Collection 62
3.8.4 Using DTrace on Java Applications 64
Chapter 4 Disk Behavior and Analysis 67
4.1 Terms for Disk Analysis 67
4.2 Random vs. Sequential I/O 69
4.2.1 Demonstration of Sequential Disk Activity 69
4.2.2 Demonstration of Random Disk Activity 69
4.3 Storage Arrays 70
4.4 Sector Zoning 71