Optimal Resource Allocation in Federated Metacomputing

204 Pages·2010·4.47 MB·English

WARSAW UNIVERSITY OF
TECHNOLOGY
Faculty of Electronics and Information
Technology
Ph.D. THESIS
Paweł Rubach, M.Sc.
Optimal Resource Allocation in Federated Metacomputing
Environments
Supervisor
Professor Kazimierz Subieta, Ph.D., D.Sc.
Warsaw, 2010
© Copyright by Paweł Rubach, Warsaw 2010
[email protected]
To my family and friends

Streszczenie
Federacyjne środowiska metaobliczeniowe zapewniają klientom możliwość dynamicznego
wywoływania usług oferowanych przez współpracujących usługodawców należących do
wirtualnej sieci usług. Tworzą one nową warstwę abstrakcji ponad pojedynczymi
komputerami, gridami czy też chmurą (ang. cloud). W odróżnieniu od wymienionych wyżej
koncepcji, warstwa ta zmienia paradygmat tworzenia programów poprzez wprowadzenie idei
metaprogramów, czyli programów w terminach innych programów. Systemy operacyjne
zostały opracowane jako warstwa pośrednia pomiędzy sprzętem a programami użytkownika.
W podobny sposób, federacyjne środowiska metaobliczeniowe ewoluują w kierunku
systemów metaoperacyjnych, które stanowią warstwę pośrednią między metaprogramami a
wirtualnym metakomputerem, który składa się ze współpracujących ze sobą usługodawców.
Jednym z kluczowych zadań każdego systemu operacyjnego jest zarządzanie zasobami.
Obecne federacyjne środowiska metaobliczeniowe nie spełniają tej roli i wobec tego nie jest
możliwe optymalne przydzielanie usługodawców do żądań klientów ani zapewnienie
odpowiedniej niezawodności ze względu na brak gwarancji dotyczących poziomu jakości
usługi (ang. SLA: Service Level Agreement). W tej pracy zdefiniowano moduł zarządzania
zasobami dla systemu metaoperacyjnego oraz przedstawiono nowe, oparte na gwarancjach
SLA, środowisko metaobliczeniowe zwane SERViceable Metacomputing Environment
(SERVME), które umożliwia przydzielanie zadań do usługodawców za pomocą m.in.
parametrów QoS (ang. Quality of Service) oraz zapewnia autonomiczne zarządzanie
dostępnością usługodawców (uruchamianie i zatrzymywanie na żądanie) w zależności od
potrzeb klientów. Rozprawa ta proponuje w szczególności architekturę, komponenty, model
obiektowy SLA, a także algorytmy związane z negocjacją SLA oraz koordynacją
wykonywania metaprogramów. Opisane eksperymenty przeprowadzone na prototypie
potwierdzają, że SERVME pozwala na znacząco efektywniejsze wykonywanie
metaprogramów.
Słowa kluczowe: metaobliczenia, środowiska rozproszone, zarządzanie zasobami QoS, SLA,
negocjacja SLA, architektura zorientowana usługowo, system metaoperacyjny
5
Abstract
Federated metacomputing environments make available to requestors the ability to
dynamically invoke services offered by collaborating providers in the virtual service network.
They create a new layer of abstraction on top of current grids, clouds and single computer
platforms, one that in contrast to aforementioned solutions, introduces a new paradigm that
changes the way programs are written. This new paradigm is the concept of metaprograms –
programs written in terms of other programs. For single computer platforms, operating
systems were introduced as an intermediary layer between the hardware and the user's
applications. Similarly, federated metacomputing environments evolve into metaoperating
systems that form an intermediary layer between metaprograms and the virtual metacomputer
composed of collaborating services. One of the crucial roles of any operating system is
resource management. Current federated metacomputing environments do not address this
issue and, consequently the assignment of providers to customer’s requests cannot be
optimized and cannot offer high reliability without relevant SLA guarantees. This dissertation
defines a resource management module of a metaoperating system by proposing a new SLA-
based SERViceable Metacomputing Environment (SERVME) that is capable of matching
providers based on QoS requirements and performing on-demand provisioning and
deprovisioning of services according to dynamic requestor's needs. In particular, the
dissertation defines the architecture and the components, introduces a QoS/SLA model, and
proposes algorithms for SLA negotiation and execution of metaprograms with SLA
guarantees. The validation of a prototype of the proposed solution shows that the system leads
to significant optimization of the execution of metaprograms.
Keywords: Metacomputing, Distributed Computing, Resource Management, QoS, SLA, SLA
negotiation, Service-Oriented Architecture, Service Object-Oriented Architecture,
Metaoperating System.

6
Acknowledgments
I would like to express my gratitude to my thesis director, Professor Kazimierz Subieta, for
his invaluable advice and help from the very first moment when I approached him as a
confused student, with no clear idea of where I was heading. He supported me in my efforts to
obtain a Fulbright fellowship that changed my academic life completely. This endeavor was
successful and I spent a year at Texas Tech University, Texas, USA, where I met Professor
Michał Sobolewski. At that time, I was searching in the dark, trying to find inspiration in
various domains of computer science, but none of the ideas I was exploring was satisfactory.
It was Professor Sobolewski who infused creative life into my work. He showed me that it
was possible to combine innovative theoretical work with cutting edge practical applications
in the field of concurrent engineering. My pessimism turned into enthusiasm.
I will always remember the first visit in Professor Sobolewski's office in Lubbock,
which lasted over three hours. During this appointment and innumerable later appointments,
we discussed not only the details of my project but many issues of general nature that helped
enormously to shape my computer science world view. At times, the discussions would take
us beyond our professional field into the realm of philosophy. His inspiring criticism of my
work and his attachment to detail and perfection challenged me to the utmost.
I would like to thank my professors, instructors and colleagues at the Warsaw School
of Economics, where I learned about the fundamentals in my field of specialization. It was so
reassuring to know that I could always count on getting good advice and as much help as I
needed.
I was greatly inspired by an exchange of ideas with Dennis Reedy, the architect and
creator of the Rio Project, and Raymond M. Kolonay, Principal Aerospace Engineer at the Air
Force Research Laboratory in Dayton, Ohio, USA. Dennis Reedy introduced me to the
architecture of the Rio provisioning framework that was an invaluable resource of ideas for
my work. His advice was instrumental in the integration of the Rio framework with the
prototype of the proposed solution. Ray Kolonay led me through the maze of optimization
techniques and algorithms and helped me identify the research methods that were appropriate
for my work.
7
I am also grateful to the Orange Labs, particularly to Andrzej Tucholski and Emil
Kowalczyk for much fruitful discussion and invaluable technical help with finding the
equipment that was crucial for running my experiments.
I would also like to thank my friends, Iga Korneta, Jerzy Orłowski and Jacek Jońca-
Jasiński who helped me find real world material to run my tests and validate my hypotheses.
Last but not least, I would like to thank my family and all those who supported me at
every stage in my work, from the inception to the completion of this dissertation.

8
Table of Contents
Introduction...............................................................................................................................17
1.1. Problem statement........................................................................................................19
1.2. Dissertation outline.......................................................................................................20
Chapter 2. Background and Literature Review.........................................................................23
2.1. Evolution of the computing platform...........................................................................24
2.2. Operating systems .......................................................................................................25
2.3. Developments of the processor....................................................................................27
2.3.1. Supercomputing and High Performance Computing............................................27
2.3.2. Distributed vs. parallel computing........................................................................28
2.3.3. Virtual processors.................................................................................................29
2.4. Grid computing.............................................................................................................31
2.4.1. Grid computing technologies................................................................................33
2.4.1.1. Globus Toolkit...............................................................................................34
2.4.1.2. UNICORE.....................................................................................................34
2.4.1.3. gLite .............................................................................................................36
2.4.1.4. Sun Grid Engine............................................................................................37
2.4.1.5. Legion...........................................................................................................38
2.4.2. Critical discussion on grid computing..................................................................38
2.5. Metacomputing.............................................................................................................40
2.6. The metacomputing platform.......................................................................................43
2.6.1. 8 fallacies of distributed computing.....................................................................44
2.6.2. Jini........................................................................................................................44
2.6.3. SORCER and metaprogramming.........................................................................46
2.6.4. Service messaging and exertions.........................................................................49
2.6.5. SORCER metacomputing platform......................................................................50
2.7. Resource management in distributed computing.........................................................52
2.7.1. QoS/SLA model....................................................................................................54
2.7.2. SLA negotiation....................................................................................................55
2.7.3. Project Rio............................................................................................................56
9
2.8. Optimal resource allocation and management.............................................................57
2.9. Summary......................................................................................................................58
Chapter 3. Requirements Analysis............................................................................................59
3.1. Social metaphor of SERVME.......................................................................................59
3.2. Functional and technical requirements.........................................................................61
3.3. Use case analysis.........................................................................................................62
3.3.1. Use cases related to the execution of an exertion.................................................62
3.3.2. Use cases related to the administration of SLAs..................................................72
3.4. Use case scenario from the real world..........................................................................75
3.5. Summary......................................................................................................................76
Chapter 4. Architecture and Design..........................................................................................77
4.1. Methodology, modeling tools and approaches............................................................77
4.1.1. Methodology.........................................................................................................78
4.1.2. Commonality variability analysis.........................................................................79
4.1.3. Cohesive design....................................................................................................80
4.1.4. UML diagrams......................................................................................................80
4.2. Conceptual architecture................................................................................................80
4.3. QoS/SLA model...........................................................................................................83
4.3.1. Basic structures: QosContext................................................................................84
4.3.2. Decision to abandon SLA negotiation scenarios in QosContext..........................91
4.3.3. Service Level Agreements contract model: SlaContext.......................................92
4.4. Architecture – components and interfaces ...................................................................96
4.4.1. SERVME component architecture........................................................................96
4.5. Interactions within the framework ............................................................................100
4.5.1. Simple exertions: tasks.......................................................................................101
4.5.1.1. Simplified flow of the SLA negotiation......................................................101
4.5.1.2. Detailed flow: preliminary selection of providers......................................102
4.5.1.3. Negotiation..................................................................................................103
4.5.1.4. SLA acceptance and signing.......................................................................106
4.5.1.5. On-demand provisioning............................................................................106
4.5.1.6. SLA monitoring and management..............................................................107
4.5.1.7. Deprovisioning services..............................................................................108

10