High Performance Inter-kernel Communication and Networking in a Replicated-kernel Operating ...

90 Pages·2016·1.76 MB·English

High Performance Inter-kernel Communication and Networking in
a Replicated-kernel Operating System
BMSaifAnsary
ThesissubmittedtotheFacultyofthe
VirginiaPolytechnicInstituteandStateUniversity
inpartialfulfillmentoftherequirementsforthedegreeof
MasterofScience
in
ComputerEngineering
BinoyRavindran,Chair
AntonioBarbalace
RobertoPalmieri
September21,2015
Blacksburg,Virginia
Keywords: Filesystem,HeterogeneousArchitecture,MessagingLayer,High-performance
Networking
Copyright2015,BMSaifAnsary
High Performance Inter-kernel Communication and Networking in a
Replicated-kernel Operating System
BMSaifAnsary
(ABSTRACT)
Modern computer hardware platforms are moving towards high core-count and heterogeneous In-
structionSetArchitecture(ISA)processorstoachieveimprovedperformanceassinglecoreperfor-
mance has reached its performance limit. These trends put the current monolithic SMP operating
system (OS) under scrutiny in terms of scalability and portability. Proper pairing of computing
workloads with computing resources has become increasingly arduous with traditional software
architecture.
One of the most promising emerging operating system architectures is the Multi-kernel. Multi-
kernelsnotonlyaddressscalabilityissues,butalsoinherentlysupportheterogeneity. Furthermore,
provide an easy way to properly map computing workloads to the correct type of processing re-
sourcesinpresenceofheterogeneity. Multi-kernelsdosobypartitioningtheresourcesandrunning
independentkernelinstancesandco-operatingamongstthemselvestopresentaunifiedviewofthe
systemtotheapplication. Popcornisonethemostprominentmulti-kernelstoday,whichisunique
inthesensethatitrunsmultipleLinuxinstancesondifferentcoresorgroupofcores,andprovides
aunifiedviewofthesystemi.e.,SingleSystemImage(SSI).
This thesis presents four contributions. First, it introduces a filesystem for Popcorn, which is a
vital part to provide a SSI. Popcorn supports thread/process migration that requires migration of
file descriptors which is not provided by traditional filesystems as well as popular distributed file
systems, this work proposes a scalable messaging based file descriptor migration and consistency
protocolforPopcorn.
Second, multi-kernel OSs rely heavily on a fast low latency messaging layer to be scalable. Mes-
saging is even more important in heterogeneous systems where different types of cores are on
different islands with no shared memory. Thus, another contribution proposes a fast-low latency
messaging layer to enable communication among heterogeneous processor islands for Heteroge-
neousPopcorn.
With advances in networking technology, newest Ethernet technologies are able to support up to
40 Gbps bandwidth, but due to scalability issues in monolithic kernels, the number of connec-
tions served per second does not scale with this increase in speed.Therefore, the third and fourth
contributions try to address this problem with Snap Bean, a virtual network device and Angel, an
opportunisticloadbalancerforPopcorn’snetworksystem.
With the messaging layer Popcorn gets over 30% performance benefit over OpenCL and Intel
Offloadingtechnique(LEO)[4]. AndwithNetPopcornweachieveover7to8timesbetterperfor-
manceovervanillaLinuxand2to5timesoverstate-of-the-artAffinityAccept[39].
Acknowledgments
I am incredibly grateful to all the people who helped in this endeavor, I would like to thank the
followingpeoplefromthebottomofmyheart:
Dr. Binoy Ravindran, for providing me direction and guidance. His inspiration made me more
focusedtowardsmywork.
Dr. AntonioBarbalace,forhisexpertopinionsandwizardrywithLinuxkernel.
Dr. RobertoPalmieri,forhiskindadvicesandinspirations.
Marina Sadini for her wonderful, cheerful presence even in the most tensed situations and help
withNetPopcorn.
YuzhongWenforhishelpwiththemessaginglayer.
Dr. Sachine Hirve, Akshay Ravichandrand, Ajitchandra Saya, Sharath Bhat, Rob Lyerly, Christo-
pherJelesnianskiforbeingwonderfulcolleagues.
AllmyfriendsinVirginiaTechcricketteamforgivingmesuchawonderfultime.
iii
Contents
1 Introduction 1
1.1 ResearchContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 ThesisOrganization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Relatedworks 6
2.1 FileSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 NetworkFileSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 KerrighedDistributedFileSystem . . . . . . . . . . . . . . . . . . . . . . 7
2.1.3 HareFileSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.4 BarrelfishFileSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.5 Non-cacheCoherentFileSystems . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Inter-kernelMessaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.1 MessagePassingConcepts . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.2 MessagePassinginCommodityMulticoreSystems . . . . . . . . . . . . . 10
2.2.3 DifferentNotificationMechanism . . . . . . . . . . . . . . . . . . . . . . 10
2.2.4 HardwareExtensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 AdvancedNetworkingSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.1 AffinityAccept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.2 Pika . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.3 IX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
iv
CONTENTS CONTENTS
2.3.4 Sandstorm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.5 NetworkStackParallelization . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4 Multi-kernelOperatingSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4.1 Barrelfish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.2 FactoredOperatingSystem . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.3 Corey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.4 TwinLinux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3 PopcornLinuxArchitecture 19
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.2 BasicArchitectureofPopcorn . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3 PopcornServices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.3.1 ProcessMigration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.3.2 ThreadMigration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3.3 AddressSpaceMigration . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3.4 PageConsistencyProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.3.5 PopcornNamespaceService . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3.6 POSIXSignals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3.7 Inter-ThreadSynchronization . . . . . . . . . . . . . . . . . . . . . . . . 23
3.4 HomogeneousandHeterogeneousPopcorn . . . . . . . . . . . . . . . . . . . . . 23
4 FileSystem 25
4.1 BootingwithIdenticalRootDirectory . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2 FileDescriptorMigration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.2.1 FileDescriptorsinLinux . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.2.2 FileDescriptorMigrationMechanism . . . . . . . . . . . . . . . . . . . . 27
4.2.3 FileDescriptorconsistencyProtocol . . . . . . . . . . . . . . . . . . . . . 27
4.3 ExtensionofLinuxVirtualFileSystemLayer . . . . . . . . . . . . . . . . . . . . 33
v
CONTENTS CONTENTS
4.3.1 RemoteOperationsImplementation . . . . . . . . . . . . . . . . . . . . . 33
4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5 MessagingLayer 35
5.1 IntelXeon-XeonPhiHeterogeneousSystem . . . . . . . . . . . . . . . . . . . . . 35
5.1.1 TheSystemSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.1.2 SymmetricCommunicationsInterface . . . . . . . . . . . . . . . . . . . . 36
5.1.3 SCIFConcepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.2 PopcornMessagingLayer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.3 SingleChannelPopcornMessaginglayer . . . . . . . . . . . . . . . . . . . . . . 41
5.3.1 DesignandImplementationofSingleChannelMessagingLayer . . . . . . 41
5.4 Multi-ChannelMessagingLayer . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.5 BufferedMessagingLayer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6 NetPopcorn: AHighlyScalableSoftwareNetworkStack 48
7 SnapBean: AReplicated-KernelDeviceDriverModelForMultiqueueDevices 52
7.1 SnapBean: Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.2 SnapBean: Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.2.1 Tx/RxfromSecondarykernels . . . . . . . . . . . . . . . . . . . . . . . . 54
7.2.2 InterruptRouting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7.2.3 FrameworkforLoadBalancing . . . . . . . . . . . . . . . . . . . . . . . 57
7.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
8 Angel: AnOpportunisticLoadBalancer 59
8.1 ConnectionsStatistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.2 LoadRedistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
8.3 FlowGroupMigration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
8.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
vi
CONTENTS CONTENTS
8.4.1 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
8.4.2 Re-routingofflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
8.4.3 SYN-FINHeuristic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.5 ImportantStatistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.6.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.6.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8.7 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.7.1 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.7.2 OverheadofAngel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
8.8 AsymmetryUnbalance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
8.9 AttackBalance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
9 ConcludingRemarksAndFuturework 73
9.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
9.1.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
9.2 FutureDirections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Bibliography 76
vii
List of Figures
1.1 Linuxnetworkingperformanceonvaryingnumberofcores. . . . . . . . . . . . . 3
2.1 OverviewofthekDFSsystem[33] . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Haredesign.copiedfrom[21] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Operatingsystemgradient[44] . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1 BasicarchitectureofPopcornLinux . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.1 BasicarchitectureofPopcornLinuxFilesystem . . . . . . . . . . . . . . . . . . . 26
4.2 Threadmigratingwithoutfiledescriptor . . . . . . . . . . . . . . . . . . . . . . . 28
4.3 Intreactionbetweenoriginkernelandremotekernel . . . . . . . . . . . . . . . . . 28
4.4 Remotefileopenoperation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.1 Messaging layer (or communication layer) stitching the kernels together to work
asasinglesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.2 Hardwaresetupinthesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.3 SCIFconnectionestablishment[25] . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.4 DesignofSingleChannelLayer . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.5 Latencyofthemessagepassing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.6 DesignofMulti-channelMessagingLayer . . . . . . . . . . . . . . . . . . . . . . 44
5.7 PerformanceofMessagingLayeronvaryingcoresandN . . . . . . . . . . . . . . 44
5.8 PerformanceofISAwithvaryingnumberofthreads . . . . . . . . . . . . . . . . . 45
5.9 PerformanceofISBwithvaryingnumberthreads . . . . . . . . . . . . . . . . . . 45
viii
LISTOFFIGURES LISTOFFIGURES
5.10 PerformanceofISCwithvaryingnumberthreads . . . . . . . . . . . . . . . . . . 45
5.11 DesignofBufferedMessagingLayer . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.12 PerformanceofIS.Awithvaryingnumberofthreads . . . . . . . . . . . . . . . . 46
5.13 PerformanceofIS.Bwithvaryingnumberthreads . . . . . . . . . . . . . . . . . . 46
5.14 PerformanceofIS.Cwithvaryingnumberthreads . . . . . . . . . . . . . . . . . . 46
5.15 Popcorncomparedwithothermechanisms . . . . . . . . . . . . . . . . . . . . . . 47
6.1 Connectionsacceptedpersecond . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.2 Breakdownoflocksinapacheserverinstance . . . . . . . . . . . . . . . . . . . . 50
6.3 Breakdownofapplications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.1 Popcornproxynetworkdevice . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.2 SnapBeannetworkdevice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.3 MSIxaddressformat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7.4 MSIxDataFormat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7.5 SnapBeanvsixgbesimplemircobenchmark . . . . . . . . . . . . . . . . . . . . 57
7.6 SnapBeanvsixgbeapachebenchmark . . . . . . . . . . . . . . . . . . . . . . . 58
8.1 Apachewebserver,numberofrequestspersecond. . . . . . . . . . . . . . . . . . 64
8.2 Apachewebserver,numberoffailedconnections. . . . . . . . . . . . . . . . . . . 64
8.3 Apachewebserver,timetoservearequest. . . . . . . . . . . . . . . . . . . . . . . 64
8.4 Lighttpdwebserver,numberofrequestspersecond. . . . . . . . . . . . . . . . . . 64
8.5 Lighttpdwebserver,numberoffailedconnections. . . . . . . . . . . . . . . . . . . 64
8.6 Lighttpdwebserver,timetoservearequest. . . . . . . . . . . . . . . . . . . . . . 64
8.7 Lighttpdwebserver,additionalrequestspersecondservedduetoAngel. . . . . . . 66
8.8 Lighttpdwebserver,additionalfailedconnectionsduetotheAngel. . . . . . . . . . 66
8.9 Lighttpdwebserver,withandwithoutAngel,timeperrequestcomparison. . . . . . 66
8.10 Lighttpdwebserver,responsetimeprofileinthecasealltheflowgroupsareonone
kernelwithoutAngel(Unbalanced),andwithAngelon8Cores . . . . . . . . . . 67
8.11 Lighttpdwebserver,responsetimeprofileinthecasealltheflowgroupsareonone
kernelwithoutAngel(Unbalanced),andwithAngelon16Cores . . . . . . . . . . 67
ix
LISTOFFIGURES LISTOFFIGURES
8.12 Lighttpdwebserver,responsetimeprofileinthecasealltheflowgroupsareonone
kernelwithoutAngel(Unbalanced),andwithAngelon24Cores . . . . . . . . . . 68
8.13 Asymmetricassignment. Popcornwith/withoutAngel. Lighttpdrequestspersecond. 68
8.14 Asymmetricassignment. Popcornwith/withoutAngel. Lighttpdfailedconnections. 68
8.15 Asymmetricassignment. Popcornwith/withoutAngel. Lighttpdtimeperrequest. . 68
8.16 ExecutiontraceofLighttpdwebserveronPopcornwithandwithoutloadbalancer
for8,16,and24CPUs. Allflowgroupsareonasinglekernelandtherequestsare
homogeneouslydistributed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.17 Lighttpdwebserver,timeperrequestprofilesattackingonekernelwith8totalkernels 70
8.18 Lighttpdwebserver,timeperrequestprofilesforprofilesattackingonekernelwith
16totalkernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
8.19 Lighttpdwebserver,timeperrequestprofilesprofilesattackingonekernelwith23
totalkernels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
8.20 PopcornvsAffinity. Requestspersecond. . . . . . . . . . . . . . . . . . . . . . . 70
8.21 PopcornvsAffinity. Failedconnections . . . . . . . . . . . . . . . . . . . . . . . 70
8.22 PopcornvsAffinity. Timeperrequest . . . . . . . . . . . . . . . . . . . . . . . . 70
8.23 Execution trace of Lighttpd webserver comparing Popcorn and Affinity on 8, 16,
and 23 CPUs. All flow groups are homogeneously distributed but requests are
initiallydirectedtoonekernelonly. . . . . . . . . . . . . . . . . . . . . . . . . . 72
9.1 PossibleDesignwithfilesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
x