1.2.1 Windows API

1048 Pages·2009·18.74 MB·English

Table of Contents
Table of Contents..............................................................................i

1. Concepts and Tools .....................................................................1

1.1 Windows Operating System Versions...........................................................1
1.2 Foundation Concepts and Terms...................................................................2
1.2.1 Windows API.........................................................................................2
1.2.2 Services, Functions, and Routines.......................................................4
1.2.3 Processes, Threads, and Jobs...............................................................4
1.2.4 Virtual Memory...................................................................................14
1.2.5 Kernel Mode vs. User Mode...............................................................17
1.2.6 Terminal Services and Multiple Sessions..........................................20
1.2.7 Objects and Handles...........................................................................21
1.2.8 Security................................................................................................22
1.2.9 Registry................................................................................................23
1.2.10 Unicode...............................................................................................24
1.3 Digging into Windows internals...................................................................24
1.3.1 Reliability and Performance Monitor...............................................25
1.3.2 Kernel Debugging...............................................................................27
1.3.3 Windows Software Development Kit................................................32
1.3.4 Windows Driver Kit............................................................................32
1.3.5 Sysinternals Tools................................................................................32
1.4 Conclusion.....................................................................................................33
2. System Architecture..................................................................34

2.1 Requirements and Design Goals..................................................................34
2.2 Operating System Model..............................................................................35
2.3 Architecture Overview..................................................................................36
2.3.1 Portability............................................................................................38
2.3.2 Symmetric Multiprocessing...............................................................39
2.3.3 Scalability.............................................................................................44
2.3.4 Differences Between Client and Server Versions.............................44
2.3.5 Checked Build.....................................................................................48
2.4 Key System Components..............................................................................50
2.4.1 Environment Subsystems and Subsystem DLLs..............................51
2.4.2 Ntdll.dll................................................................................................57
2.4.3 Executive..............................................................................................58
2.4.4 Kernel...................................................................................................61
2.4.5 Hardware Abstraction Layer.............................................................64
2.4.6 Device Drivers.....................................................................................67
2.4.7 System Processes.................................................................................72
i
2.5 Conclusion.....................................................................................................83
3. System Mechanisms ..................................................................84

3.1 Trap Dispatching...........................................................................................84
3.1.1 Interrupt Dispatching.........................................................................86
3.1.2 Exception Dispatching......................................................................113
3.1.3 System Service Dispatching.............................................................123
3.2 Object Manager..........................................................................................134
3.2.1 Executive Objects..............................................................................137
3.2.2 Object Structure................................................................................138
3.3 Synchronization...........................................................................................171
3.3.1 High-IRQL Synchronization............................................................172
3.3.2 Low-IRQL Synchronization.............................................................177
3.4 System Worker Threads.............................................................................196
3.5 Windows global Flags.................................................................................198
3.6 Advanced Local Procedure Calls (ALPCs)...............................................201
3.7 Kernel event Tracing..................................................................................205
3.8 Wow64..........................................................................................................210
3.8.1 Wow64 Process Address Space Layout...........................................211
3.8.2 System Calls.......................................................................................211
3.8.3 Exception Dispatching......................................................................212
3.8.4 User Callbacks...................................................................................212
3.8.5 File System Redirection....................................................................212
3.8.6 Registry Redirection and Reflection...............................................213
3.8.7 I/O Control Requests......................................................................214
3.8.8 16-Bit Installer Applications............................................................214
3.8.9 Printing............................................................................................214
3.8.10 Restrictions....................................................................................215
3.9 user-Mode Debugging.................................................................................215
3.9.1 Kernel Support..................................................................................215
3.9.2 Native Support..................................................................................217
3.9.3 Windows Subsystem Support..........................................................218
3.10 Image Loader............................................................................................218
3.10.1 Early Process Initialization............................................................220
3.10.2 Loaded Module Database...............................................................221
3.10.3 Import Parsing................................................................................224
3.10.4 Post Import Process Initialization.................................................226
3.11 Hypervisor (Hyper-V)...............................................................................226
3.11.1 Partitions..........................................................................................228
3.11.2 Root Partition..................................................................................228
3.11.3 Child Partitions...............................................................................230
3.11.4 Hardware Emulation and Support................................................232
3.12 Kernel Transaction Manager...................................................................237
3.13 Hotpatch Support......................................................................................239
ii
3.14 Kernel Patch Protection...........................................................................241
3.15 Code integrity............................................................................................244
3.16 Conclusion.................................................................................................245
4. Management Mechanisms ......................................................246

4.1 The Registry................................................................................................246
4.1.1 Viewing and Changing the Registry................................................246
4.1.2 Registry Usage...................................................................................246
4.1.3 Registry Data Types..........................................................................247
4.1.4 Registry Logical Structure...............................................................248
4.1.6 Monitoring Registry Activity...........................................................258
4.1.7 Registry Internals..............................................................................261
4.2 Services.........................................................................................................275
4.2.1 Service Applications..........................................................................276
4.2.2 The Service Control Manager..........................................................291
4.2.3 Service Startup..................................................................................293
4.2.4 Startup Errors...................................................................................297
4.2.5 Accepting the Boot and Last Known Good.....................................298
4.2.6 Service Failures.................................................................................300
4.2.7 Service Shutdown..............................................................................300
4.2.8 Shared Service Processes..................................................................302
4.2.9 Service Tags.......................................................................................304
4.2.10 Service Control Programs..............................................................305
4.3 Windows Management instrumentation...................................................306
4.3.1 Providers............................................................................................308
4.3.2 The Common Information Model and the Managed Object Format
Language.....................................................................................................309
4.3.3 Class Association...............................................................................311
4.3.4 WMI Implementation.......................................................................314
4.3.5 WMI Security....................................................................................315
4.4 Windows Diagnostic infrastructure...........................................................316
4.4.1 WDI Instrumentation.......................................................................316
4.4.2 Diagnostic Policy Service..................................................................316
4.4.3 Diagnostic Functionality...................................................................317
4.5 Conclusion...................................................................................................319
5. Processes, Threads, and Jobs .................................................320

5.1 Process Internals.........................................................................................320
5.1.1 Data Structures..................................................................................320
5.1.2 Kernel Variables................................................................................327
5.1.3 Performance Counters......................................................................327
5.1.4 Relevant Functions............................................................................328
5.2 Protected Processes.....................................................................................330
5.3 Flow of CreateProcess................................................................................332
iii
5.3.1 Stage 1: Converting and Validating Parameters and Flags..........333
5.3.2 Stage 2: Opening the Image to Be Executed...................................334
5.3.3 Stage 3: Creating the Windows Executive Process Object
(PspAllocate- Process)...............................................................................337
5.3.4 Stage 4: Creating the Initial Thread and Its Stack and Context..341
5.3.5 Stage 5: Performing Windows Subsystem–Specific
Post-Initialization.......................................................................................342
5.3.6 Stage 6: Starting Execution of the Initial Thread..........................344
5.3.7 Stage 7: Performing Process Initialization in the Context of the New
Process.........................................................................................................344
5.4 Thread Internals.........................................................................................351
5.4.1 Data Structures..................................................................................351
5.4.2 Kernel Variables................................................................................358
5.4.3 Performance Counters......................................................................358
5.4.4 Relevant Functions............................................................................359
5.4.5 Birth of a Thread...............................................................................359
5.5 Examining Thread activity.........................................................................360
5.6 Worker Factories (Thread Pools)..............................................................364
5.7 Thread Scheduling......................................................................................368
5.7.1 Overview of Windows Scheduling...................................................368
5.7.2 Priority Levels...................................................................................370
5.7.3 Windows Scheduling APIs................................................................372
5.7.4 Relevant Tools...................................................................................372
5.7.5 Real-Time Priorities..........................................................................374
5.7.6 Thread States.....................................................................................375
5.7.7 Dispatcher Database.........................................................................378
5.7.8 Quantum............................................................................................379
5.7.9 Scheduling Scenarios........................................................................386
5.7.10 Context Switching...........................................................................390
5.7.11 Idle Thread.......................................................................................390
5.7.12 Priority Boosts.................................................................................391
5.7.13 Multiprocessor Systems..................................................................404
5.7.14 Multiprocessor Thread-Scheduling Algorithms...........................412
5.7.15 CPU Rate Limits.............................................................................413
5.8 Job Objects..................................................................................................414
5.9 Conclusion...................................................................................................419
6. Security.....................................................................................420

6.1 Security Ratings..........................................................................................420
6.2 Security System Components.....................................................................422
6.3 Protecting Objects.......................................................................................425
6.3.1 Access Checks....................................................................................426
6.3.2 Security Descriptors and Access Control........................................448
6.4 Account Rights and Privileges...................................................................462
iv
6.4.1 Account Rights..................................................................................463
6.4.2 Privileges............................................................................................464
6.4.3 Super Privileges.................................................................................468
6.5 Security Auditing........................................................................................469
6.6 logon.............................................................................................................471
6.6.1 Winlogon Initialization.....................................................................472
6.6.2 User Logon Steps...............................................................................474
6.7 User account Control..................................................................................478
6.7.1 Virtualization.....................................................................................478
6.7.2 Elevation............................................................................................484
6.8 Software Restriction Policies.....................................................................489
6.9 Conclusion...................................................................................................490
7. I/O System................................................................................491

7.1 I/O System Components.............................................................................491
7.2 Device Drivers.............................................................................................495
7.2.1 Types of Device Drivers....................................................................495
7.2.2 Structure of a Driver.........................................................................501
7.2.3 Driver Objects and Device Objects.................................................503
7.2.4 Opening Devices................................................................................508
7.3 I/O Processing..............................................................................................514
7.3.1 Types of I/O........................................................................................514
7.3.2 I/O Request to a Single-Layered Driver.........................................523
7.3.3 I/O Requests to Layered Drivers.....................................................529
7.3.4 I/O Cancellation................................................................................537
7.3.5 I/O Completion Ports........................................................................541
7.3.6 I/O Prioritization...............................................................................546
7.3.7 Driver Verifier...................................................................................551
7.4 Kernel-Mode Driver Framework (KMDF)..............................................553
7.4.1 Structure and Operation of a KMDF Driver..................................553
7.4.2 KMDF Data Model...........................................................................555
7.4.3 KMDF I/O Model..............................................................................559
7.5 user-Mode Driver Framework (uMDF)....................................................562
7.6 The Plug and Play (PnP) Manager............................................................565
7.6.1 Level of Plug and Play Support.......................................................566
7.6.2 Driver Support for Plug and Play...................................................567
7.6.3 Driver Loading, Initialization, and Installation.............................569
7.6.4 Driver Installation.............................................................................578
7.7 The Power Manager....................................................................................582
7.7.1 Power Manager Operation...............................................................584
7.7.2 Driver Power Operation...................................................................585
7.7.3 Driver and Application Control of Device Power..........................589
7.8 Conclusion...................................................................................................589
8. Storage Management ..............................................................591

v
8.1 Storage Terminology...................................................................................591
8.2 Disk Drivers.................................................................................................592
8.2.1 Winload..............................................................................................592
8.2.2 Disk Class, Port, and Miniport Drivers..........................................592
8.2.3 Disk Device Objects..........................................................................596
8.2.4 Partition Manager.............................................................................597
8.3 Volume Management..................................................................................598
8.3.1 Basic Disks.........................................................................................598
8.3.2 Dynamic Disks...................................................................................600
8.3.3 Multipartition Volume Management...............................................605
8.3.4 The Volume Namespace....................................................................615
8.3.5 Volume I/O Operations.....................................................................621
8.3.6 Virtual Disk Service..........................................................................623
8.4 BitLocker Drive encryption.......................................................................625
8.4.1 BitLocker Architecture.....................................................................625
8.4.2 Encryption Keys................................................................................626
8.4.3 Trusted Platform Module (TPM)....................................................628
8.4.4 BitLocker Boot Process....................................................................630
8.4.5 BitLocker Key Recovery..................................................................631
8.4.6 Full Volume Encryption Driver.......................................................632
8.4.7 BitLocker Management....................................................................633
8.5 Volume Shadow Copy Service...................................................................633
8.5.1 Shadow Copies..................................................................................634
8.5.2 VSS Architecture...............................................................................634
8.5.3 VSS Operation...................................................................................635
8.5.4 Uses in Windows................................................................................637
8.6 Conclusion...................................................................................................642
9. Memory Management.............................................................643

9.1 Introduction to the Memory Manager......................................................643
9.2 Services the Memory Manager Provides..................................................647
9.2.1 Large and Small Pages.....................................................................648
9.2.2 Reserving and Committing Pages...................................................649
9.2.3 Locking Memory...............................................................................650
9.2.4 Allocation Granularity......................................................................651
9.2.5 Shared Memory and Mapped Files.................................................651
9.2.6 Protecting Memory...........................................................................654
9.2.7 No Execute Page Protection.............................................................655
9.2.8 Copy-on-Write...................................................................................659
9.2.9 Address Windowing Extensions.......................................................661
9.3 Kernel-Mode Heaps (System Memory Pools)..........................................662
9.3.1 Pool Sizes............................................................................................663
9.3.2 Monitoring Pool Usage.....................................................................665
9.3.3 Look-Aside Lists................................................................................668
vi
9.4 Heap Manager.............................................................................................669
9.4.1 Types of Heaps...................................................................................670
9.4.2 Heap Manager Structure..................................................................670
9.4.3 Heap Synchronization......................................................................671
9.4.4 The Low Fragmentation Heap.........................................................672
9.4.5 Heap Security Features....................................................................673
9.4.6 Heap Debugging Features................................................................674
9.4.7 Pageheap............................................................................................674
9.5 Virtual address Space layouts....................................................................675
9.5.1 x86 Address Space Layouts..............................................................676
9.5.2 x86 System Address Space Layout..................................................679
9.5.3 x86 Session Space..............................................................................679
9.5.4 System Page Table Entries...............................................................682
9.5.5 64-Bit Address Space Layouts..........................................................684
9.5.6 64-Bit Virtual Addressing Limitations............................................686
9.5.7 Dynamic System Virtual Address Space Management..................689
9.5.8 System Virtual Address Space Quotas............................................692
9.5.9 User Address Space Layout..............................................................693
9.6 Address Translation....................................................................................697
9.6.1 x86 Virtual Address Translation......................................................697
9.6.2 Translation Look-Aside Buffer........................................................703
9.6.3 Physical Address Extension (PAE)..................................................704
9.6.4 IA64 Virtual Address Translation...................................................707
9.6.5 x64 Virtual Address Translation......................................................708
9.7 Page Fault Handling...................................................................................709
9.7.1 Invalid PTEs......................................................................................710
9.7.2 Prototype PTEs.................................................................................711
9.7.3 In-Paging I/O.....................................................................................713
9.7.4 Collided Page Faults.........................................................................714
9.7.5 Clustered Page Faults.......................................................................714
9.7.6 Page Files...........................................................................................715
9.8 Stacks............................................................................................................719
9.9 Virtual address Descriptors.......................................................................721
9.10 NuMa..........................................................................................................724
9.11 Section Objects..........................................................................................725
9.12 Driver Verifier...........................................................................................732
9.13 Page Frame Number Database................................................................736
9.13.1 Page List Dynamics.........................................................................739
9.13.2 Page Priority....................................................................................740
9.13.3 Modified Page Writer.....................................................................743
9.13.4 PFN Data Structures.......................................................................744
9.14 Physical Memory limits............................................................................748
9.15 Working Sets..............................................................................................752
9.15.1 Demand Paging...............................................................................753
vii
9.15.2 Logical Prefetcher...........................................................................753
9.15.3 Placement Policy.............................................................................757
9.15.4 Working Set Management..............................................................757
9.15.5 Balance Set Manager and Swapper...............................................760
9.15.6 System Working Set........................................................................761
9.15.7 Memory Notification Events..........................................................762
9.16 Proactive Memory Management (SuperFetch)......................................764
9.16.1 Components.....................................................................................765
9.16.2 Tracing and Logging.......................................................................766
9.16.3 Scenarios..........................................................................................767
9.16.4 Page Priority and Rebalancing......................................................768
9.16.5 Robust Performance.......................................................................770
9.16.6 ReadyBoost......................................................................................771
9.16.7 ReadyDrive......................................................................................772
9.17. Conclusion................................................................................................774
10. Cache Manager......................................................................775

10.1 Key Features of the Cache Manager.......................................................775
10.2 Cache Virtual Memory Management......................................................779
10.3 Cache Size..................................................................................................780
10.4 Cache Data Structures..............................................................................784
10.4.1 Systemwide Cache Data Structures...............................................784
10.4.2 Per-File Cache Data Structures.....................................................786
10.5 File System interfaces...............................................................................791
10.5.1 Copying to and from the Cache.....................................................792
10.5.2 Caching with the Mapping and Pinning Interfaces.....................793
10.5.3 Caching with the Direct Memory Access Interfaces....................795
10.6 Fast I/O......................................................................................................796
10.7 read ahead and Write behind...................................................................798
10.7.1 Intelligent Read-Ahead...................................................................798
10.7.2 Write-Back Caching and Lazy Writing........................................799
10.7.3 Write Throttling..............................................................................805
10.7.4 System Threads...............................................................................807
10.8 Conclusion.................................................................................................808
11. File Systems............................................................................809

11.1 Windows File System Formats.................................................................810
11.2 File System Driver architecture...............................................................815
11.2.1 Local FSDs.......................................................................................815
11.2.2 Remote FSDs....................................................................................816
11.2.3 File System Operation.....................................................................819
11.2.4 File System Filter Drivers...............................................................824
11.3 Troubleshooting File System Problems...................................................825
11.4 Common log File System..........................................................................827
viii
11.5 NTFS Design goals and Features.............................................................834
11.5.1 High-End File System Requirements............................................834
11.5.2 Advanced Features of NTFS..........................................................835
11.6 NTFS File System Driver.......................................................................847
11.7 NTFS On-Disk Structure.......................................................................850
11.8 NTFS recovery Support............................................................................882
11.8.1 Design...............................................................................................883
11.8.2 Metadata Logging...........................................................................884
11.8.3 Recovery...........................................................................................888
11.8.4 NTFS Bad-Cluster Recovery..........................................................891
11.8.5 Self-Healing......................................................................................894
11.9 Encrypting File System Security..............................................................895
11.9.1 Encrypting a File for the First Time..............................................898
11.9.2 The Decryption Process..................................................................902
11.9.3 Backing Up Encrypted Files...........................................................903
11.10 Conclusion................................................................................................904
12. Networking.............................................................................905

12.1 Windows Networking Architecture.........................................................905
12.1.1 The OSI Reference Model..............................................................905
12.1.2 Windows Networking Components...............................................907
12.2 Networking APIs.......................................................................................909
12.2.1 Windows Sockets.............................................................................910
12.2.2 Winsock Kernel (WSK)..................................................................915
12.2.3 Remote Procedure Call...................................................................917
12.2.4 Web Access APIs.............................................................................921
12.2.5 Named Pipes and Mailslots............................................................923
12.2.6 NetBIOS...........................................................................................928
12.2.7 Other Networking APIs..................................................................930
12.3 Multiple redirector Support.....................................................................934
12.3.1 Multiple Provider Router...............................................................934
12.3.2 Multiple UNC Provider..................................................................937
12.4 Name resolution........................................................................................938
12.5 Location and Topology.............................................................................941
12.6 Protocol Drivers........................................................................................943
12.7 NDiS Drivers..............................................................................................952
12.7.1 Variations on the NDIS Miniport..................................................955
12.7.2 Connection-Oriented NDIS............................................................956
12.7.3 Remote NDIS...................................................................................958
12.7.4 QoS...................................................................................................959
12.8 Binding.......................................................................................................961
12.9 Layered Network Services.......................................................................962
12.10 Conclusion.............................................................................................967
13. Startup and Shutdown..........................................................968

ix