Operating system from 0 to 1

309 Pages·2022·6.009 MB·English

by  Do Hoang Tu

T U , D O H OA N G
O P E R A T I N G S Y S T E M S :
F R O M 0 T O 1

Contents
Preface i
I Preliminary 1
1 Domain documents . . . . . . . . . . . . . . . 3
1.1 Problem domains . . . . . . . . . . . . . . . 3
1.2 Documents for implementing a problem domain . . . . 6
1.3 Documents for writing an x86 Operating System. . . . 9
2 From hardware to software: Layers of abstraction . . 11
2.1 The physical implementation of a bit. . . . . . . . 11
2.2 Beyond transistors: digital logic gates . . . . . . . 12
2.3 Beyond Logic Gates: Machine Language . . . . . . 17
2.4 Abstraction . . . . . . . . . . . . . . . . . 26
3 Computer Architecture . . . . . . . . . . . . . 33
3.1 What is a computer? . . . . . . . . . . . . . 33
3.2 Computer Architecture . . . . . . . . . . . . . 39
3.3 x86 architecture . . . . . . . . . . . . . . . 44
3.4 Intel Q35 Chipset. . . . . . . . . . . . . . . 47
3.5 x86 Execution Environment . . . . . . . . . . . 47
II tu, do hoang
4 x86 Assembly and C . . . . . . . . . . . . . . 49
4.1 objdump . . . . . . . . . . . . . . . . . . 50
4.2 Reading the output . . . . . . . . . . . . . . 51
4.3 Intel manuals . . . . . . . . . . . . . . . . 53
4.4 Experiment with assembly code . . . . . . . . . . 54
4.5 Anatomy of an Assembly Instruction. . . . . . . . 56
4.6 Understand an instruction in detail . . . . . . . . 66
4.7 Example: jmp instruction . . . . . . . . . . . . 69
4.8 Examine compiled data. . . . . . . . . . . . . 72
4.9 Examine compiled code. . . . . . . . . . . . . 86
5 The Anatomy of a Program . . . . . . . . . . . 107
5.1 Reference documents: . . . . . . . . . . . . . 109
5.2 ELF header . . . . . . . . . . . . . . . . . 109
5.3 Section header table . . . . . . . . . . . . . . 114
5.4 Understand Section in-depth. . . . . . . . . . . 121
5.5 Program header table . . . . . . . . . . . . . 141
5.6 Segments vs sections. . . . . . . . . . . . . . 144
6 Runtime inspection and debug . . . . . . . . . . 151
6.1 A sample program . . . . . . . . . . . . . . 151
6.2 Static inspection of a program . . . . . . . . . . 152
6.3 Runtime inspection of a program . . . . . . . . . 163
6.4 How debuggers work: A brief introduction . . . . . . 179
II Groundwork 191
7 Bootloader . . . . . . . . . . . . . . . . . . 193
7.1 x86 Boot Process . . . . . . . . . . . . . . . 193
7.2 Using BIOS services. . . . . . . . . . . . . . 194
7.3 Boot process. . . . . . . . . . . . . . . . . 195
operating systems: from 0 to 1 III
7.4 Example Bootloader . . . . . . . . . . . . . . 195
7.5 Compile and load. . . . . . . . . . . . . . . 196
7.6 Loading a program from bootloader . . . . . . . . 201
7.7 Improve productivity with scripts . . . . . . . . . 205
8 Linking and loading on bare metal. . . . . . . . . 217
8.1 Understand relocations with readelf . . . . . . . . 218
8.2 Crafting ELF binary with linker scripts . . . . . . . 227
8.3 C Runtime: Hosted vs Freestanding . . . . . . . . 248
8.4 Debuggable bootloader on bare metal . . . . . . . . 249
8.5 Debuggable program on bare metal. . . . . . . . . 251
III Kernel Programming 275
9 x86 Descriptors. . . . . . . . . . . . . . . . . 277
9.1 Basic operating system concepts . . . . . . . . . 277
9.2 Drivers . . . . . . . . . . . . . . . . . . 279
9.3 Userspace and kernel space . . . . . . . . . . . 279
9.4 Memory Segment . . . . . . . . . . . . . . . 280
9.5 Segment Descriptor . . . . . . . . . . . . . . 280
9.6 Types of Segment Descriptors . . . . . . . . . . 280
9.7 Descriptor Scope . . . . . . . . . . . . . . . 280
9.8 Segment Selector . . . . . . . . . . . . . . . 280
9.9 Enhancement: Bootloader with descriptors. . . . . . 280
10 Process . . . . . . . . . . . . . . . . . . . . 281
10.1 Concepts . . . . . . . . . . . . . . . . . . 281
10.2 Process . . . . . . . . . . . . . . . . . . 281
10.3 Threads . . . . . . . . . . . . . . . . . . 283
10.4 Task: x86 concept of a process . . . . . . . . . . 284
10.5 Task Data Structure. . . . . . . . . . . . . . 284
IV tu, do hoang
10.6 Process Implementation . . . . . . . . . . . . 284
10.7 Milestone: Code Refactor . . . . . . . . . . . . 285
11 Interrupt . . . . . . . . . . . . . . . . . . . 287
12 Memory management . . . . . . . . . . . . . . 289
13 File System . . . . . . . . . . . . . . . . . . 291
Index. . . . . . . . . . . . . . . . . . . . . 293
Biblography . . . . . . . . . . . . . . . . . . 295
Preface
Greetings!
You’ve probably asked yourself at least once how an operating system
is written from the ground up. You might even have years of program-
ming experience underyourbelt,yet yourunderstanding of operating
systems may still be a collection of abstract concepts not grounded in
actual implementation. To those who’ve never built one,an operating
system may seem like magic: a mysterious thing that can control hard-
ware while handling a programmer’s requests via the API of theirfavorite
programming language. Learning how to build an operating system seems
intimidating and difficult; no matter how much you learn, it never feels
like you know enough. You’re probably reading this book right now to
gain a betterunderstanding of operating systems to be a bettersoftware
engineer.
If that is the case, this book is for you. By going through this book,
you will be able to find the missing pieces that are essential and enable
you to implementyourown operating system from scratch! Yes,from
scratch,without going through any existing operating system layer to
prove to yourself that you are an operating system developer. You may
ask,“Isn’t it more practical to learn the internals of Linux?”.
Yes...
and no.
Learning Linux can help your workflow at your day job. However, if
you follow that route, you still won’t achieve the ultimate goal of writ-
ing an actual operating system. By writing your own operating system,
you will gain knowledge that you will not be able to glean just from learn-
ii tu, do hoang
ing Linux.
Here’s a list of some benefits of writing your own OS:
(cid:3) You will learn how a computer works at the hardware level, and you
will learn to write software to manage that hardware directly.
(cid:3) You will learn the fundamentals of operating systems, allowing you
to adapt to any operating system, not just Linux
(cid:3) To hack on Linux internals suitably, you’ll need to write at least one
operating system on your own. This is just like applications program-
ming: to write a large application,you’ll need to start with simple ones.
(cid:3) You will open pathways to various low-level programming domains
such as reverse engineering, exploits, building virtual machines, game
console emulation and more. Assembly language will become one of
your most indispensable tools for low-level analysis. (But that does
not mean you have to write your operating system in Assembly!)
(cid:3) Writing an operating system is fun!
Why another book on Operating Systems?
There are many books and courses on this topic made by famous profes-
sors and experts out there already. Who am I to write a book on such
an advanced topic? While it’s true that many quality resources exist, I
find them lacking. Do any of them show you how to compile yourC code
and the C runtime library independent of an existing operating system?
Mostbooks on operating system design andimplementation only dis-
cuss the software side; how the operating system communicates with the
hardware is skipped. Important hardware details are skipped, and it’s
difficult for a self-learner to find relevant resources on the Internet. The
aim of this book is to bridge that gap: not only will you learn how to pro-
gram hardware directly,but also how to read official documents from hard-
ware vendors to program it. You no longer have to seek out resources to
help yourself interpret hardware manuals and documentation: you can
do ityourself. Lastly,I wrote this bookfrom an autodidact’s perspec-
tive. I made this book as self-contained as possible so you can spend more
operating systems: from 0 to 1 iii
time learning and less time guessing or seeking out information on the
Internet.
One of the core focuses of this book is to guide you through the pro-
cess of reading official documentation from vendors to implement your
software. Official documents from hardware vendors like Intel are criti-
cal for implementing an operating system or any other software that di-
rectly controls the hardware. At a minimum, an operating system devel-
oper needs to be able to comprehend these documents and implement
software based on a set of hardware requirements. Thus, the first chap-
ter is dedicated to discussing relevant documents and their importance.
Another distinct feature of this book is that it is “Hello World” cen-
tric. Most examples revolve around variants of a “Hello World” program,
which will acquaint you with core concepts. These concepts must be learned
before attempting to write an operating system. Anything beyond a sim-
ple “Hello World” example gets in the way of teaching the concepts,thus
lengthening the time spent on getting started writing an operating sys-
tem.
Let’s dive in. With this book, I hope to provide enough foundational
knowledge that will open doors for you to make sense of other resources.
This book will be beneficial to students who’ve just finished their first
C/C++ course greatly. Imagine how cool it would be to show prospec-
tive employers that you’ve already built an operating system.
Prerequisites
(cid:3) Basic knowledge of circuits
– Basic Concepts of Electricity: atoms,electrons,proton,neutron,cur-
rent flow.
– Ohm’s law
If you are unfamiliar with these concepts, you can quickly learn them
here: http://www.allaboutcircuits.com/textbook/,by reading chap-
ter 1 and chapter 2.
(cid:3) C programming. In particular:
iv tu, do hoang
– Variable and function declarations/definitions
– While and for loops
– Pointers and function pointers
– Fundamental algorithms and data structures in C
(cid:3) Linux basics:
– Know how to navigate directory with the command line
– Know how to invoke a command with options
– Know how to pipe output to another program
(cid:3) Touch typing. Since we are going to use Linux, touch typing helps. I
know typing speed does not relate to problem-solving,but at least your
typing speed should be fast enough not to let it get in the way and de-
grade the learning experience.
In general,I assume that the readerhas basic C programming knowledge,
and can use an IDE to build and run a program.
What you will learn in this book
(cid:3) How to write an operating system from scratch by reading hardware
datasheets. In the real world, you will not be able to consult Google
for a quick answer.
(cid:3) Write code independently. It’s pointless to copy and paste code. Real
learning happens when you solve problems on yourown. Some exam-
ples are provided to help kick start your work, but most problems are
yours to conquer. However, the solutions are available online for you
after giving a good try.
(cid:3) A big picture of how each layer of a computer related to each other,
from hardware to software.
(cid:3) How to use Linux as a development environment and common tools
for low-level programming.
(cid:3) How a program is structured so that an operating system can run.