Secure Content-Based Routing Using Intel Software Guard Extensions

0.5 MB·

by  Rafael Pires

Secure Content-Based Routing Using
Intel Software Guard Extensions
Rafael Pires Marcelo Pasin Pascal Felber
UniversityofNeuchâtel UniversityofNeuchâtel UniversityofNeuchâtel
[e-mail] [e-mail] [e-mail]
Christof Fetzer
TUDresden
[e-mail]
7
1 ABSTRACT 1. INTRODUCTION
0
2 Content-based routing (CBR) is a powerful model that sup- Content-based routing (CBR) is a flexible and powerful
portsscalableasynchronouscommunicationamonglargesets paradigmforscalablecommunicationamongdistributedpro-
n of geographically distributed nodes. Yet, preserving privacy cesses. It decouples data producers from consumers, and
a represents a major limitation for the wide adoption of CBR, dynamically routes messages based on their content. While
J
notably when the routers are located in public clouds. In- the publish/subscribe communication model has been ex-
7 deed, a CBR router must see the content of the messages tensively studied over more than a decade [11], and many
1 sent by data producers, as well as the filters (or subscrip- implementationshavebeenproposed(e.g.,[26,7,19]),itstill
tions)registeredbydataconsumers. Thisrepresentsamajor fails to reach wide deployment and usage in the era of cloud
C] deterrent for companies for which data is a key asset, as for computing.
instance in the case of financial markets or to conduct sen- One of the major reasons to the lack of general adoption
D
sitive business-to-business transactions. While there exists can be tracked down to privacy concerns. Indeed, CBR re-
. some techniques for privacy-preserving computation, they quiresroutercomponentstofiltermessagesbymatchingtheir
s
c are either prohibitively slow or too limited to be usable in content against a (potentially large) collection of subscrip-
[ real systems. In this paper, we follow a different strategy tions that act as a reverse index and, hence, must be stored
by taking advantage of trusted hardware extensions that by the filtering engines. In turn, this requires the router to
1
have just been introduced in off-the-shelf processors and see the content of both the messages and the subscriptions,
v
provide a trusted execution environment. We exploit Intel’s which represents a major threat for companies for which
2
new software guard extensions (SGX) to implement a CBR data is a key asset. For instance, in the emblematic exam-
1
engine in a secure enclave. Thanks to the hardware-based ple of financial trading, stock quotes published by exchange
6
trustedexecutionenvironment(TEE),thecompute-intensive platforms have commercial value and must be protected,
4
0 CBR operations can operate on decrypted data shielded by while subscriptions may reveal sensitive information about a
. the enclave and leverage efficient matching algorithms. Ex- client’s portfolio and must also be secured.
1 tensive experimental evaluation shows that SGX adds only Several approaches to supporting privacy-preserving pub-
0 limited overhead to insecure plaintext matching outside se- lish/subscribe have been proposed recently [20, 25], but
7 cure enclaves while providing much better performance and noneprovidesatthesametimepowerfulfilteringcapabilities
1 morepowerfulfilteringcapabilitiesthanalternativesoftware- and high performance. Approaches that rely on sophisti-
:
v only solutions. To the best of our knowledge, this work is cated cryptographic techniques like (fully) homomorphic
i the first to demonstrate the practical benefits of SGX for encryption can support a wide range of operations but are
X
privacy-preserving CBR. prohibitively slow—several orders of magnitude slower than
r plaintext filtering. Performance can be improved by exploit-
a
Keywords ing specialised techniques like asymmetric scalar-product
preserving encryption (ASPE) [7], but these approaches suf-
Content-based routing, publish/subscribe, security, privacy, fer from severe limitations in the type of operations they
SGX. support, which is typically restricted to equality matching
or degradedforms of range queries. Furthermore, these tech-
niquesalsooftenintroduceadditionaloverheads,e.g.,ASPE’s
space complexity grows exponentially with the number of
Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalor attributes.
classroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributed
In order to combine both security and performance, one
forprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitation
onthefirstpage. Copyrightsforcomponentsofthisworkownedbyothersthanthe can instead resort on hardware-based secure execution envi-
author(s)mustbehonored.Abstractingwithcreditispermitted.Tocopyotherwise,or ronments. However, such extensions have been so far lim-
republish,topostonserversortoredistributetolists,requirespriorspecificpermission
ited to narrow, domain-specific applications (e.g., military)
and/[e-mail].
and unsuitable for deployment in cloud environments, no-
Middleware’16Trento,Italy
tably because of their high cost and dependency on custom
(cid:13)c 2016Copyrightheldbytheowner/author(s). PublicationrightslicensedtoACM.
ISBN978-1-4503-4300-8/16/12...$15.00 hardware. This is about to change thanks to availability
DOI:http://dx.doi.org/10.1145/2988336.2988346
of trusted execution environments (TEEs) in commodity Instructions
pSscoukrfotyteclwaecaksrsreyeopmrtgsoiu.cgarIrornadaprphecahxirctitateielnclcyust-iluosarinrge,snspeu(rdSpoGpacpoeXsrpst)loi,acrtasnhtebiaowutnisaeltxlilntouewsanissnsiegoucnsuIenrsretesetel,t’nocscanlelalxeevwede- Application environment SErGunnXct liaumvseeer ApplicatSiorRGnuunXntt iiummseeer EEEEEEGRERXNEEEITPSTTKEOUREMRYTE
ECREATE
tcbpheeesaTrspftohorrrieos.mcfiDseahrsnassitceteaelddSioeGsivdneeXrnfp-rhcleoaernmyainadpbtt)oeletexidhndteswifrpdohrcreiomoletcdheoe(esuasrentousnrdniscdnlheahievnatneghvc.eeeoenswnhctiithplhaepveaesd,lammbiounesttplcarnatonoe- Privileged ed environment OSP astgreu ctatubrleeRsuntSimGeXH amrdowdaurele EEEEEEEAEIBTWLNDRXDLBITOATDECCNKKD
2015 in a mobile version, and desktop and server variants xposHW Platform EEPRAEMOVE
have followed in 2016. Hence this technology is very new E
and still largely untested in real-world settings. In this pa-
Figure 1: SGX enclaves are a trusted execution en-
per, weintroduceanoriginalCBRarchitecturethatexploits
vironment for unprivileged applications. Enclaves
the SGX technology to execute a routing engine in a secure
can only be created with the help of the operat-
enclave. Weproposeaprotocolforsecurelyexchangingcryp-
ing system. In this way, the operating system can
tographic keys between data producers and consumers, and
control which applications are permitted to create
the SGX-protected routing engine. Both publications and
enclaves as well as the number and the size of the
subscriptions are encrypted and signed, thus protecting the
enclaves.
system from unauthorised parties observing or tampering
with the information. Our system, called SCBR, thus com-
bines a key exchange protocol and a state-of-the-art routing
different jurisdiction. Moreover, from a technical perspec-
engine to provide both security and performance while exe-
tive, operating systems and hypervisor encompass millions
cutingundertheprotectionofthesecureenclave. Tothebest
of lines of source code. The number of exploitable bugs
ofourknowledge,SCBRisthefirstsystemtoexperimentally
is proportional to the number of lines of code. Hence, on
evaluate and demonstrate the benefits of executing CBR in
shared infrastructures, one cannot neglect the probability of
a TEE.
applications being attacked by other tenants’ applications
In this paper, we propose the following contributions. We
executing on the same computer. With formal proofs of
first present a detailed description of the SGX technology
the operating system [16], one might be able to reduce the
and how it can be used to develop secure implementation
number of exploitable bugs. Yet, the system administrators
(§2). We then introduce an original secure CBR architecture
of the cloud provider still have access to all application data
thatcombinesoperationsonencrypteddataoutsideenclaves,
and, therefore, their credentials are often hijacked to gain
and efficient matching of publications and subscriptions in-
access to systems and all application data being processed
side enclaves (§3). We conduct in-depth evaluation on a
[28].
real implementation with several workloads to observe the
One needs a technology that protects the confidentiality
sources of performance overheads and the various trade-offs
and integrity of application data from access by any other
ofSGX,andweprovidecomparativeresultsagainstplaintext
software, even software with higher privileges like the hy-
(insecure) matching as well as an ASPE-based alternative
pervisor. Trusted execution environments (TEE) are a way
(§4). The paper is completed by a discussion of related work
to protect the integrity and confidentiality of data. ARM
(§5) and concluding remarks (§6).
TrustZone [1] is a popular trusted execution environment
that provides a secure world that cannot be accessed by the
2. BACKGROUND
normal world. It provides, however, only one secure world.
Hence, applications either need to share the secure world, or
2.1 IntelSoftwareGuardExtensions(SGX)
at most one application can use the secure world. Moreover,
Traditionally,oneprotectstheintegrityandconfidentiality the secure world is under the control of a separate operating
ofapplicationsbyenforcingtheisolationofapplications. An system, which means that it can still be accessed by some
operating system isolates the applications using hardware system administrator.
mechanisms like virtual address spaces and privileged in- Intel SGX is a recently released technology that provides
structions. Multiple operating systems running on the same TEEs implemented by the CPU (see Figure 1). Although
physical host are isolated by the hypervisor using hardware TEE solutions have been proposed and implemented pre-
virtualisation extensions provided by the CPU. viously, SGX has the advantage that each application can
The protection of the application integrity and confiden- create separate secure enclaves to protect the confidentiality
tiality requires so far that both the operating system as well and integrity of its data while it is being processed. This
asthehypervisoraretrusted. Incloudenvironments,atleast closesthegapoftraditionalapproachesthatprotectthedata
the hypervisor and, in some cases, also the operating system usingprotocolsliketransportlayersecurity(TLS)[8]during
are under the control of the cloud provider. Trusting the transmission, and using file encryption when storing it—but
hypervisor and the operating system in cloud environments so far do not protect it while it is processed.
can raise legal and technical issues since the cloud provider Theprotectionofdataduringprocessingisparticularlyim-
is a different legal entity from the application provider. portantinthecontextofcloudcomputing,whereapplications
Fromalegalpointofview,contractualagreementswiththe run in an environment under the control of a different legal
cloud provider could ensure legal protection of the data in- entity. Developers can ensure the end-to-end confidentiality
tegrity and confidentiality. However, such agreements might and integrity of their application’s data by terminating the
be insufficient in case the cloud provider is located in a TLSconnectionsthatconnecttheapplicationwithitsclients
TEE (SGX) it was dirty or just dropped if a copy already exists in main
Untrusted code
memory. After a page is evicted, the SGX driver loads the
Trusted code
Call requested page (that triggered the fault) from main memory.
gate
Trusted function The SGX driver closely interacts with the CPU for eviction
Create enclave Execute and paging since the CPU keeps track of the authentication
tagsoftheevictedpagesandchecksthemwhenloadingpages
Call trusted function Return into the EPC. In this way, one does not need to trust the
SGXdriversinceitcannotviolatetheconfidentialitynorthe
integrity of enclave pages.
…
Enclave Confidentiality in the traffic between CPU and system
memory is achieved by a component called the memory en-
cryption engine (MEE), which is also responsible for provid-
Figure 2: Sensitive data can be protected inside ing tamper resistance and replay protection. Under normal
of an enclave. This data can be accessed by calling processoroperation,memorytransactionsthatmissthecache
trusted functions inside of the enclave. are handled by the memory controller (MC). If however the
cache miss is translated to a protected region, MEE takes
over. Inthiscase,itencryptsordecryptsdatabeforesending
toorfetchingfromsystemmemory,inadditiontoperforming
inside of an enclave, and any data at rest remains encrypted
integrity checks [12].
inside the enclave. With this approach, neither the cloud
Memory checks are made through an integrity tree [10]
provider nor any hacker with root access can compromise
thatusesastatefulmessageauthenticationcode(MAC)with
the integrity or confidentiality of application data.
nonces [22] (non repeating numbers, coined to be used only
The basic idea underlying SGX is that an application can
once). The tree is stored in untrusted memory, except for
keep its confidential data inside of an enclave and access
the root that is kept on-die and inaccessible from outside. It
it from within (see Figure 2). For example, an enclave
reflects the integrity of the whole protected area at a given
could protect the credentials of an external database and
time, thus precluding any attacks by means of modification
additionally ensure that only certain queries are issued, or
or replay of values in memory. Any mismatch during a
limit the maximum frequency of the database queries sent
verification causes a MC lock, which ultimately requires a
by this application.
machine reboot [12].
For small applications, SGX can protect application data
TheSGXsystemprovideseachenclavewithasealkey that
even when the attacker has physical access to the computer.
canbeusedtostoredataonstablestorageandaccessitagain
ThehardwaresecurityperimeteristheCPUpackage,andall
upon subsequent execution. This facilitates the development
data belonging to an enclave is encrypted and authenticated
of applications that can restart an enclave without requiring
when stored in main memory. External snooping, such as
a new remote attestation. The enclave instead loads its
eavesdropping the memory bus or the system memory itself,
secrets from a configuration file encrypted with the enclave-
will hence not reveal any data stored inside of enclaves.
specific seal key and kept in stable storage. Note that an
All data read from memory is checked for integrity and
attacker could still try to serve an enclave with a previous
freshnessbyverifyingtheauthenticationtags. Externaldata
version of a configuration file that is properly encrypted and
modifications are therefore detected since the authentication
authenticated. To prevent such replay attacks, an enclave
codes will not match. Feeding the CPU with old data that
can use the monotonic counter facilities provided by the
was properly encrypted and authenticated is also detected
platform. Each time an enclave writes a new version of its
by keeping track of the latest authentication codes for each
configuration data, it increments a monotonic counter and
page.
stores the new value inside the configuration file. When the
To enable an application to use enclaves, the developer
enclave restarts, it reads the monotonic counter and checks
must provide a signed shared library (.so or .dll) that will
that it matches the values stored in the configuration file.
execute inside anenclave. The library itself is not encrypted
and can be inspected before being started, hence no secret
3. SCBRARCHITECTURE
should be stored inside the code.
An enclave is provided with secrets, like certificates and We describe in this section the overall architecture of
keys, with the help of a remote attestation protocol. This SCBR.Wefirstexplaintherationalethatdrovethedesignof
protocol can prove that an enclave runs on a genuine Intel’s thesystem. Wethenintroducetheconsideredmodel,andwe
processorwithSGXandverifythatitsidentitymatchesthat presenttheprincipleofsubscriptionregistrationandmessage
ofthecodethatthedeveloperaskedtostart[12]. Duringre- publication. We finally discuss some implementation details.
moteattestation,asecurechannelisestablishedthatpermits
3.1 ObjectivesandAssumptions
the remote entity to provide the enclave with secrets.
Enclave code and data are stored in a memory area pre- Designing a secure, privacy-preserving CBR system is not
defined at boot time, called the enclave page cache (EPC), trivial, even if one can rely on trusted hardware. Consider
which is at the moment limited to 128MB. Applications the example of a stock exchange publishing streams of finan-
can use approximately 90MB while the remaining space is cial data (stock quotes). Consumers who want to protect
reserved for SGX itself. If an enclave is larger than 90MB, their confidential data (portfolio) have to trust the code of
any access to an enclave page that does not reside in the the CBR engine, which typically originates from the stock
EPCresultsinapagefault. Thepagefaultishandledbyan exchange. They may also want to restrict the ability to see
SGXdriverintheoperatingsystemthatselectsapageofthe their subscriptions to a single publisher, and not other data
EPCtoevict,i.e.,thepageismovedtomainmemoryincase providers that leverage the same software and infrastructure
Service (data) provider Producer Consumer
➊
Source … Source … Source PK-1 SK
Publication
➋
➌
Enclave Enclave ➍ SK ➏
… ➎
Enclave
TEE (SGX) TEE (SGX)
Infrastructure provider
TEE (SGX)
Publication Subscription
Figure 4: Interaction amongst entities.
Client … Client … Client
Figure 3: Components and roles. Messagesarecomposedofapayload,whichisofinterestto
theendusersbutopaquetotheSCBRsystem,andaheader
that contains several attributes and associated values. The
(e.g., different stock exchange operators). CBRenginesfiltermessagesbasedontheattributevaluesin
We therefore consider a simple model based on the fol- their header.
lowing assumptions. The publish/subscribe system operates Subscriptions are composed of predicates specifying con-
as a service under the control of a single“service provider” straints over the attributes. Predicates can include equality
thatpublishesdata. Consumersaretheclientsoftheservice constraintsorgenerallyanykindofrangesoverthevaluesof
and typically pay recurring fees to have access to the data. the attributes. For instance, a subscriber interested in spe-
Hence,theremustbewaysfortheproducerstocontrolwhich cificquotesforacompanywhenitreachesacertainpricecan
subscribers can join and read data from the service, and to registerasubscriptionsuchas“symbol="HAL"∧price<50”.
exclude clients that stop paying their fees or behave in a We say that a message matches a subscription if its header
non-trustworthy manner. The publishers operate within the satisfies the constraints expressed in the subscription predi-
administrative domain of the service provider from which cate.
data originates, and they are trusted by the clients for the Subscriptions are typically stored by the CBR engine in
purpose of the considered service. This model closely maps, a dedicated data structure that operates as an“inverted”
for instance, to the aforementioned scenario of the stock database. By exploiting relationships between the different
exchange. predicates,aspioneeredinSiena[5],onecanbothreducethe
Given the trust relationships between the different com- memory footprint of the subscription index and improve the
ponents of the system, it appears clear that publishers and matching speed. In particular, the property of containment
clients must share cryptographic keys that are not known (also called covering) can be leveraged to avoid unnecessary
by the infrastructure. Furthermore, there should be some tests. Essentially, we say that a subscription s contains or
lightweight mechanisms for the publishers to (dis-)allow covers another subscription s(cid:48), denoted by s (cid:119) s(cid:48), if any
clients from accessing new data, independently of whether event that matches s(cid:48) also matches s. That is, s is more
they had been legitimate customers in the past. general than s(cid:48). For instance, predicate“x>0”covers both
predicates“x=1”and subscription“x>0∧y =1”. Note
3.2 SystemModel
that the containment relationship creates a partial order
Following the considerations discussed above, we distin- on subscriptions. In SCBR, we use a matching algorithm
guish three main roles in our architecture, as illustrated in that exploits containment to minimise the footprint of the
Figure 3: subscriptions stored in the enclave, where only a limited
amount of memory is available.
• The service provider (or data provider) produces in- SCBRmakesuseofbothsymmetricandasymmetric(pub-
formation flows for the clients, typically“as a service” lic key) cryptography. The former is more efficient and is
and for a fee. The data may be produced by mul- used for communication between the publishers and the en-
tiple sources (publishers) operating within the same claves,whilethelatterisusedbetweenclientsandtheservice
administrative domain. provider when registering subscriptions, as will be detailed
later.
• The infrastructure provider hosts the CBR engines in
the cloud. It provides secure hardware (SGX-based
3.3 TheSubscriptionProcess
enclaves) and performs the actual data routing and
transmissionthroughitsnetwork. Asitoperatesunder Asmentionedabove,SCBRwasdesignedsothatproducers
a different administrative domain and may share its are the owners of the generated data. They have therefore
resources among several customers (in a multi-tenant theabilitytodecidewhethertheyacceptasubscriptionfrom
configuration), the infrastructure provider is trusted a client, as well as to subsequently invalidate it. To control
neither by the data provider nor by its clients. access to the service, we rely upon an additional admission
phasewhenregisteringanewsubscription. Theclientcannot
• The clients of the service are the end users who are freely submit its subscriptions to the CBR engines in the
interested in the actual data and subscribe to informa- cloud, but has to go through a data producer. Informally,
tion flows via the CBR engines. They trust the data the registration process works as follows (see Figure 4).
providers but not the infrastructure. Consideraclientcthatwantstoregisterasubscriptionsby
theroutingenginerandsubsequentlyreceivemessagemsent Workload Proportion of Number of Subs values
name equality predicates attributes distribution
by the data producer p. The publisher has a public/private
e100a1 100% : 1eq. pred.
key pair (PK/PK−1), as well as symmetric key (SK) that e80a1 8–11(original)
issharedwiththecoderunningintheenclave,butunknown e80a2 20% : 0eq. pred. 2×more
80% : 1eq. pred.
to clients and to the infrastructure provider—this is made e80a4 4×more
possible thanks to SGX, as explained in §2. extsub2 15% : 0eq. pred. 2×more Uniform
60% : 1eq. pred.
1. Theclientfirstencryptsitssubscriptionsusingthedata 15% : 2eq. pred.
extsub4 4×more
provider’s public key, hence preventing unauthorised 10% : 3eq. pred.
parties to see it, and sends the resulting encrypted e80a1z100 20% : 0eq. pred. Zipfonsymbol
e80a1zz100 80% : 1eq. pred. 8–11(original) Zipfonall
subscription {s} to p.
PK e100a1zz100 100% : 1eq. pred. attributes
2. Then, after decrypting and verifying that the subscrip-
Table 1: Workloads description (adapted from [4]).
tionisvalid,aswellasverifyingtheclient’sstatus,the
publisher re-encrypts s using SK and signs it. It then
sends the encrypted subscription {s} to the routing
SK
engine r. 3.5 ImplementationDetails
3. Finally,rvalidatesanddecryptsthesubscriptioninside Intel provides a set of tools to aid the coding and the
the enclave (remember that SK is only known to the fulfilmentofSGXrequirements. Theirsoftwaredevelopment
code running within the enclave), and inserts it in its kit(SDK)comprisesageneratorforproxiesandstubswritten
index. in C that should be linked to both trusted and untrusted
code. This generation is based on a text-based configuration
Note that the subscriptions also embed information about
filethatfollowsthesyntaxoftheenclave definition language,
the clients that it visible to the code running outside the
which basically defines the interface of the edge routines.
enclave. This allows the router to establish connexions with
Since system calls and input/output instructions are not
the consumers and forward relevant messages to them.
allowed inside secure containers, Intel also provides libraries
3.4 ThePublicationProcess that are guaranteed to comply with these limitations. We
used Intel’s STL implementation and AES-CTR encryption
Once subscriptions have been registered by the clients,
functions for enclave code.
data can be routed along the reverse path. The publication
TheSDKalsoincludestheenclavesigningtool responsible
process works as follows.
forthemeasurementandsignature(§2)ofthesharedlibrary
4. The publisher encrypts the header of the message m that will be loaded in the protected container [13]. Intel’s
using SK, which is only known to the code running in- SDK is provided both for Windows and Linux, and in this
sidetheenclave(encryptionofthepayloadisdiscussed work, we used the latter version.
below). The encrypted message {m}SK is then sent to In SCBR, encryption outside the enclave is implemented
the routing engine r. using the Crypto++ library,1 using respectively AES-CTR
5. Upon receiving the message, r decrypts the header in andRSAforsymmetricandasymmetricencryption. Weuse
the enclave, leaving the opaque payload outside, and the ZeroMQ library2 for communication, and we serialise
matchesitagainstitssubscriptionindex. Theresultof bothplain-textorencryptedmessagesinBase64textformat.
this operation is a list of clients that have registered a Information about page faults is obtained via the Linux
matching subscription. system’s getrusage function (attribute minflt). Similarly,
we rely on a Linux system call to configure and read the
6. Finally,rforwardstheencryptedmessagepayloadtoall processor’s performance counters for cache misses.
clientsthathavebeenidentifiedaspartofthematching
operation.
4. EVALUATION
The payload of messages is encrypted separately using a
We evaluated the system as described in Figure 4, with
(symmetric)groupkeysharedbetweenthepublisherandthe
boththeproducerandconsumerrunninginonemachineand
consumers. Thisallowspublisherstoperiodicallychangethe
the filtering engine in another. Measurements were collected
key as the population of customers evolves. In particular,
at the machine running the filter, which was equipped with
this enables publishers to prevent clients that have cancelled
anIntelSkylakeCPUmodeli7-6700runningat3.4GHzwith
their membership from accessing newly published messages.
an 8MB cache and 8GB of main memory. We allocated
This process is orthogonal to the encryption performed for
128MB of main memory to EPC (maximum allowed).
protectingtheheaderandsubscriptions,andusedforprivacy-
ToevaluateSCBRandtofacilitatecomparison,wereused
preserving CBR. It is hence out of the scope of this paper
theworkloadsfrompreviouswork[4]composedof9datasets.
and not discussed further.
They were built based on real data corresponding to ran-
Note that having multiple routers in the path would in-
domlyselectedstockquotesfromtheYahoo!financewebsite.3
crease the complexity of the key management between pub-
Approximately250,000entrieswerecollectedinaperiodof5
lishers and matchers. We believe that an overlay broker
years,withpublicationscomposedby8to11attributes. The
network is not the best architecture for a scalable privacy-
entries collected were used to produce synthetic subscrip-
preserving pub/sub engine, and we would rather advocate a
tiondatasetscontaininganassortmentofequalityandrange
similarstructureasinStreamHub[3]wherewespecialisethe
systemcomponentsinordertogainperformance. Insuchan 1http://www.cryptopp.com/
architecture,thecurrentpublisher-matcherkeymanagement 2http://zeromq.org/
scheme could be simply replicated. 3http://finance.yahoo.com/
300
In AES e100a1
In plain 104 e80a1
250 Out AES e80a2
Out plain e80a4
s)µ 200 s) eexxttssuubb24
ng time ( 150 g time (µ103 eee881000aa011azz11zz10z010000
chi hin
Mat 100 atc
M
50 102
0
0 10 20 30 40 50 60 70 80 90 100
Number of registered subscriptions (×103) 103 104 105
Number of subscriptions
Figure 5: Overhead of encryption and enclave. Figure 6: Performance of the containment-based al-
gorithm applied to the different workloads in plain-
text, outside enclaves.
predicates on the quotes’ attributes. Besides, subscriptions
wereselectedaccordingto(1)auniformrandomdistribution
and (2) a Zipfian law with exponent s = 1. In order to encryption). We also measured, for each workload, the per-
assess the algorithms’ performance with a greater number of formanceofourimplementationofASPE[7, 4]asabaseline
variablesanddifferentlevelsofcontainment,someworkloads forasoftware-onlyalternativethatdoesnotuseenclaves. We
were synthesised with twice and four times the number of measured only the matching step, and not the encryption or
attributesoftheoriginalpublications,bymergingdatafrom decryption of ASPE messages. The presented ASPE perfor-
multiple quotes. Table 1 summarises the characteristics of mancecostwasthereforeinherenttoitsmatchingalgorithm,
the datasets used. which grows faster than any other strategy when increasing
Our first experiment aimed at evaluating the performance the size of the subscription database. The difference is more
overhead caused by executing our filter inside an enclave. substantial for the first and last workloads, although it re-
We filled the subscription database with datasets with 1,000, mains close to at least one order of magnitude in all setups.
2,500, 5,000, 10,000, 25,000, 50,000 and 100,000 subscrip- These observations indicate that the performance penalties
tions, reaching a total memory size of approximately 43MB of SGX are largely tolerable when considering software-only
for the largest dataset. Thereafter, we sent a batch of 1,000 alternatives for secure filtering, at least when the amount of
publications to be matched against the subscriptions and memory used by the routing engine remains below the EPC
measuredthetimeittooktoaccomplisheachfilteringopera- assigned size. We will further explore this matter below.
tion. Werananidenticalsetupwithandwithoutencryption, Another interesting aspect is the gap between the curves
inside and outside an enclave, using the same filtering code. corresponding to execution inside and outside the enclaves.
When using encryption, publications and subscriptions were After approximately 10,000 subscriptions, the versions in-
encrypted in the producer and decrypted in the filter us- side and outside enclaves begin to drift apart due to the
ing AES-CTR. The average results for the first workload number of memory accesses necessary to accomplish every
(e100a1)areshowninFigure5. Byconsideringtheproximity comparison. At some point, the filtering data does not fit
of the lines with and without encryption, we can see that completelyintheprocessor’scachememoryandcachemisses
encryption overhead is small and nearly constant. Indeed, start to occur more frequently. When this happens, data
this overhead remains below 5µs both inside or outside the must be fetched from system memory and, in the case of
enclave, which is negligible when compared to the matching enclave executions, it must be decrypted and checked for
time given a reasonable large database size. The overhead integrityandfreshness. Moreover,theevictedenclave’scache
resultingfromtheenclaveismoresignificant,reachingnearly nodemustbeencryptedbeforebeingsenttosystemmemory.
40% for the largest set of subscriptions considered in our That behavior is consistent with cache miss rates (measured
experiments, which is explained by the occurence of cache outside the enclaves), which are also reported in Figure 7.
misses (cache size is shown by the vertical line). Weonlymeasuredcachemissesoutsidetheenclaves,because
We then focused on the influence of the workloads. In our Linux version failed to properly monitor the cache per-
ordertounderstandtheeffectofdifferentdatasetsonSCBR formance counters inside enclaves. Since the code running
performance, we first executed each of them without encryp- inside and outside the enclaves is the same, it is reasonable
tion outside secure enclaves. Results are shown in Figure 6. to assume that cache miss rates would be similar.
The first (e100a1) and last (e100a1zz100) workloads show We finally wanted to observe the performance penalties
thebestperformance,sinceallsubscriptionscontainequality when exceeding the maximum protected memory size and
predicatesandthesubscriptionsetformsdeepercontainment paging begins to happen. Since EPC memory is limited,
trees. In contrast, datasets with more attributes (e80a4 and whenever it is full and more space is required, pages must
extsub4)performworsebecausetheyyieldindexeswithmore be evicted from the protected area to the main (untrusted)
rootsandshallowtrees,thereforeinducingmorecomparisons memory. Accordingly, a page swap occurs every time a
to traverse the whole database. previously evicted page is accessed. Besides the fact that
Figure7displaysseparatemeasurementsforeachworkload system memory is slower than the processor’s cache, which
runningSCBRinsideandoutsideanenclave(bothusingAES already imposes performance costs, memory page swaps are
Out ASPE In AES Out AES Cache limit Cache misses (out AES) [%]
e100a1 e80a1 e80a2
105 100% 105 100% 105 100%
s)µ104 80% s)µ104 80% s)µ104 80%
me ( 60% me ( 60% me ( 60%
g ti103 g ti103 g ti103
hin 40% hin 40% hin 40%
Matc102 20% Matc102 20% Matc102 20%
101 0% 101 0% 101 0%
103 104 105 103 104 105 103 104 105
Number of registered subscriptions Number of registered subscriptions Number of registered subscriptions
e80a4 extsub2 extsub4
106 100% 106 100% 106 100%
s)µ105 80% s)µ105 80% s)µ105 80%
me (104 60% me (104 60% me ( 60%
hing ti103 40% hing ti103 40% hing ti104 40%
Matc102 20% Matc102 20% Matc103 20%
101 0% 101 0% 102 0%
103 104 105 103 104 105 103 104 105
Number of registered subscriptions Number of registered subscriptions Number of registered subscriptions
e80a1z100 e80a1zz100 e100a1zz100
105 100% 105 100% 105 100%
s)µ104 80% s)µ104 80% s)µ104 80%
me ( 60% me ( 60% me ( 60%
g ti103 g ti103 g ti103
hin 40% hin 40% hin 40%
Matc102 20% Matc102 20% Matc102 20%
101 0% 101 0% 101 0%
103 104 105 103 104 105 103 104 105
Number of registered subscriptions Number of registered subscriptions Number of registered subscriptions
Figure 7: Comparison of different approaches with varying workloads.
serviced by the operating system and hence incur an even Figure 8 shows the combined results of two executions
higher overhead. when populating the in-memory subscription storage. In
one execution we registered subscriptions inside an enclave,
and outside in the other. We used the workload e80a1 in
plaintextformat,andweexecutedthesameregistrationcode
20 40
Registration time in/out enclave in both experiments. Each point of the graph accounts for
18 (left scale) 35 an average of 5,000 points, from a set of 500,000 subscrip-
Page faults in/out enclave
16 (right scale) tions. We plotted the page fault rates observed by dividing
30 their numbers from inside and outside enclaves. The values
14
measuredoutsideareverylargeforthelargestdatabasesize,
12 25 30) reaching up to 40,000 more page faults.
Ratio 10 20 ×o (1 Wealsodividedthetimeittooktoregisteronesubscription
8 ati inside the enclave by the time required outside. We can
15 R clearlyseethepointwherepagingstartstotakeplace,when
6
memory consumption reaches just over 90MB. The vertical
10
4 line shows the EPC memory limit, which comprises both
2 5 the enclaved application memory and SGX internal data
structures. Atthemaximumsizeofourexperiment(213MB),
0 0
0 20 40 60 80 100 120 140 160 180 200 220 registering a subscription inside the enclave took 18 times
moretimethandoingitoutside. Theseresultsshowthatthe
Subscription database size (MB)
overhead grows outrageously when paging starts to happen,
andtheymakeastrongcaseforfurtherstudiesonoptimising
Figure 8: Loss in performance when exceeding EPC
the memory footprint of applications running inside secure
memory limit.
SGX enclaves.
5. RELATEDWORK based on their content. They propose architectures designed
SGXwasintroducedonlyveryrecentlyandonlyfewprac- torespondtospecificsecuritythreats,throughcreativecom-
ticalalgorithmsusingTEEshavebeenalreadyproposed. As binationsofaccesscontrolandkeymanagementmechanisms.
far as we could reach in the available literature, SCBR is Most of these solutions integrate elaborated access control
the first attempt to demonstrate the practical benefits of models into existing event-based middleware [2, 29, 26], or-
SGX for privacy-preserving CBR. This section comments ganising routing brokers in sets that share keys to encrypt
on a couple of related efforts that propose to use SGX in a or decrypt data. Different encryption granularities are used,
similar context. ranging from single attributes to the entire message. Key
IfusingSGXforCBRisnew,securecontent-basedpublish management imposes an important overhead and content-
subscribe is not. There is a vast amount of literature on the basedroutingisonlypossibleinroutersthathavethekeysto
subject,includingmajorsurveys[20,25]. Weoverviewafew decrypttherelevantmatchingattributes. Althoughtheseso-
relevant existing secure CBR systems (or families thereof) lutions isolate traffic between mutually distrusting domains,
andwediscusstheconditionsunderwhichanenclavedCBR by eventually relying on plaintext matching they discourage
version can provide an important improvement. their use in untrusted hardware as found in clouds.
In contrast to filtering with plaintext, recent research has
5.1 SGXasaSecureBuildingBlock also explored the development of specific encryption algo-
rithmsforpublish/subscribethatallowforadirectmatchof
In a recent paper, Kim et al. [15] describe three case
encrypted publications with encrypted subscriptions. One
studies that leverage TEEs in the context of networking
such solution, which we evaluated in this paper, is asym-
applications. One of them illustrated how SGX can improve
metric scalar product-preserving encryption (ASPE) [7], ini-
on privacy by running parts of software-defined network
tially introduced for secure query computation on encrypted
controllers. Another case study shows that SGX could be
databases. Publication attributes and subscriptions con-
used for protecting Tor onion routers from attacks made
straints are represented as coordinates of multidimensional
by participating nodes. The last one proposes to use SGX
points. ASPE is based on an exact relation preserving iso-
for executing in-network functions. All case studies share
morphismandsupportssubscriptioncontainment,albeititis
with SCBR the underlying idea of secure execution inside
vulnerable to known-plaintext attacks. ASPE is asymmetric,
untrusted nodes.
hence the decryption key can be shared without compromis-
VC3 [23] is a distributed, secure execution environment
ing the solution. Given that ASPE’s matching complexity is
extendingtheApacheHadoopMapReduceframework. Each
prohibitively high when using a large number of attributes,
worker node hosts an enclaved loader that runs a key ex-
researchers have proposed to enhance it with a pre-filtering
change protocol, and decrypts and executes map/reduce
approach[4]thatexpressesequalityconstraintsusingBloom
functions. VC3 guarantees global integrity by generating
filters. This allows for quickly identifying subscriptions that
work summaries within enclaved workers that can be user-
are known not to match the publication as their equality
verified after the end of a job. The proposed environment
constraint(s) cannot be satisfied.
cleanlyisolateswithinenclavesthecomputationmadeunder
Li et al. [18] proposed an approach supporting interval
a well established model. SCBR shares some design ideas
matching by transforming it into prefix-matching and us-
with VC3, but the inherently structured map/reduce com-
ing a prefix-preserving encryption algorithm that supports
munication pattern makes it unpractical for implementing
containment. The scheme has limited resistance under at-
publish/subscribe (while the converse would be possible).
tack and requires a shared key between publisher and the
5.2 SecureContent-basedPublish/Subscribe subscriber. Ion [14] subsequently devised a similar prefix-
matching algorithm derived from attribute-based encryption
Content-based publish/subscribe appears at first sight to
that presents stronger privacy guarantees but with much
be incompatible with privacy preservation, as by definition
higher encryption cost. Encryption is based on El Gamaal
messages must be filtered based on their content, i.e., the
[9], thus eliminating the need for shared keys between par-
filtering engine should be able to see both the data of publi-
ticipants (although common key material must be retrieved
cations and subscriptions. To make the problem even more
from a trusted authority).
challenging,manyfilteringtechniquesexploitstructuralprop-
Raiciu et al. [21] proposed a hybrid matching mechanism
erties of the information exchanged between publishers and
that uses different encryption techniques according to the
subscribers. In particular, by structuring the containment
types of values and constraints. It can handle partial string
relations between subscriptions, one can build efficient data
matching using a specialised bloom filter, range matching
structures to store subscriptions and match publications.
using a prefix-preserving encryption as proposed by Li et al.
Containmentallowsforasignificantreductionofthenumber
(describedabove),andafastprotocolformatchingarbitrary
of subscriptions stored as well as the number of matching
subscription functions based on Yao’s garbled circuits [27].
evaluationsexecutedperpublication. Asaconsequence,con-
The mechanism supports containment but may yield false
tainment is used in most CBR systems in use today [5, 6,
negatives. Furthermore, two identical subscriptions encrypt
17], which makes them largely incompatible with classical
to identical cyphertexts, which is semantically insecure.
cryptographic techniques for privacy preservation.
Nabeel et al. [19] proposed a solution that can be used
A recent survey suggests classifying publish/subscribe sys-
with any constraints on numerical values based on homo-
tems preserving confidentiality in two categories, depending
morphic encryption. It allows for containment, but requires
on whether they leverage traditional security techniques or
common knowledge of the parameters needed for homomor-
they rely on special-purpose forms of encryption [20].
phic encryption between publishers and subscribers and it
Solutionsbasedontraditionaltechniquesencryptsensitive
can only match numerical values.
information that traverse untrusted domains, hence basi-
Tariqetal. [24]proposedapeer-to-peerarchitecturebased
callypreventingtheinfrastructurefromfilteringpublications
on attribute-based encryption, which supports numerical the CPU. Also, we have shown that paging plays an impor-
comparisons and prefix/suffix constraints on strings. Peers tant role in reducing the enclave performance. We will work
can act as publishers or subscribers and publications are de- on optimising our data structures to avoid paging and cache
liveredthroughanoverlaystructuredasasetofcontainment- misses, by smartly storing and accessing the containment
based trees. It depends on a central and trusted authority trees, splitting them into enclaved and external parts, and
that provides master public keys for publication encryption, appropriatelyfittingthemintocachelines. Finally,thereare
and subscriptions present weak confidentiality. alsoopportunitiestooptimiseourunderlyingimplementation
All encryption schemes presented in the solutions above inordertoreducethefrequencyofenclaveenters/exits(e.g.,
are either heavier or weaker than standard production-level with OS calls). We plan to explore different strategies such
encryption (e.g., AES). By using symmetric encryption and as using message batching, implementing message exchanges
plaintext matching under trusted execution, SCBR is able at the enclave border, or even bringing more of the simple
to combine the best of both worlds. It provides a novel OS operations into the enclave itself. All in all, we expect
schemethat(i)supportsconfidentialityforeventsandfilters; that these mechanisms and optimisations will contribute to
(ii) permits publishers to express further constraints about further decrease the overhead of running inside an enclave.
who can access their events; (iii) handles filters that can
express very complex constraints on events even if brokers
are not able to access any information in clear on both
events and filters; (iv) allows brokers to use state-of-the art
7. ACKNOWLEDGMENTS
containment-based filtering, and finally (v) does not require
publishers and subscribers to share any keys. The research leading to these results has received funding
from the European Commission, Information and Communi-
6. CONCLUSION cationTechnologies,H2020-ICT-2015undergrantagreement
number 690111 (SecureCloud project). Rafael Pires is also
We presented the architecture and evaluation of SCBR, a sponsored by CNPq, National Counsel of Technological and
secure content-based routing engine that takes advantage of Scientific Development, Brazil.
thetrustedexecutionenvironmentprovidedbyshieldedSGX
enclaves, a technology available on recently released off-the-
8. REFERENCES
shelf Intel processors. In doing so, we can leverage state-of-
the-arttechniquesforefficientfilteringinplaintext,sincethe [1] ARM Limited. ARM Security Technology - Building a
trusted perimeter is limited to the CPU die. Outside that Secure System using TrustZone Technology, 2009.
boundary, private data is always encrypted and protected
[2] J. Bacon, D. M. Eyers, J. Singh, and P. R. Pietzuch.
from tampering and replay attacks, even from operating
Access control in publish/subscribe systems. In
systems,hypervisors,andadministratorswithphysicalaccess
Proceedings of the Second International Conference on
tomachines. Asaresult,wedonotsufferfromtheprohibitive
Distributed Event-based Systems, DEBS ’08, pages
performance and space overheads of software-based secure
23–34, New York, NY, USA, 2008. ACM.
approaches, such as homomorphic encryption or dedicated
[3] R.Barazzutti,P.Felber,C.Fetzer,E.Onica,J.Pineau,
algorithms like ASPE. On the other hand, the safety of our
M. Pasin, E. Rivi`ere, and S. Weigert. StreamHub: a
systemisentirelybasedonSGXanditcouldbecompromised
massively parallel architecture for high-performance
by design flaws, hardware bugs, or even trapdoors.
content-based publish/subscribe. In The 7th ACM
Aspartofourextensiveexperiments,wetestedthesystem
International Conference on Distributed Event-Based
with nine different workloads and analysed the influence of
Systems, DEBS ’13, Arlington, TX, USA - June 29 -
cache misses and page faults on the code running within
July 03, 2013, pages 63–74, 2013.
secure enclaves. While both events introduce some overhead
[4] R. Barazzutti, P. Felber, H. Mercier, E. Onica, and
(as compared to insecure matching outside the enclave), per-
E. Rivi`ere. Thrifty privacy: Efficient support for
formancedegradesmuchmoreheavilywiththelatter,which
privacy-preserving publish/subscribe. In DEBS, 2012.
occurs when exceeding the available amount of protected
[5] A.Carzaniga,D.S.Rosenblum,andA.L.Wolf.Design
memory. WealsocomparedtheperformanceofSGXagainst
and evaluation of a wide-area event notification service.
thesoftware-basedASPEalternativeandobservedthatSGX
ACM TCS, 2001.
performs systematically better as long as memory usage is
kept below 90MB. Performance deterioration when exhaust- [6] R. Chand and P. Felber. XNET: A reliable
ing memory is unsurprising. Indeed, any application that content-based publish/subscribe system. In 23rd
exceeds the EPC size limit will undergo heavy overheads International Symposium on Reliable Distributed
and scalability restrictions. This limitation can be overcome Systems (SRDS 2004), pages 264–273, 2004.
through horizontal scalability or future hardware evolutions [7] S. Choi, G. Ghinita, and E. Bertino. A
of SGX. privacy-enhancing content-based publish/subscribe
Our encouraging results open the way for further research system using scalar product preserving transformations.
on smart handling of subscriptions and matching algorithms In DEXA, Lecture Notes in Computer Science, 2010.
in order to minimise memory footprint and build an enclave- [8] T. Dierks. The Transport Layer Security (TLS)
efficient system. As future work, we intend to explore the Protocol Version 1.2. RFC 5246, Oct. 2015.
possibilities of efficiently use the enclave border and the [9] T. El Gamal. A public key cryptosystem and a
memory hierarchy. For instance, we observed that cache signature scheme based on discrete logarithms. In
misses starts to happen when we have roughly 10,000 sub- Proceedings of CRYPTO 84 on Advances in Cryptology,
scriptions, which represents 4.37MB of memory storage and pages 10–18, New York, NY, USA, 1985.
corresponds to slightly more than half of the 8MB cache in Springer-Verlag New York, Inc.
[10] R. Elbaz, D. Champagne, C. Gebotys, R. B. Lee, [20] E. Onica, P. Felber, H. Mercier, and E. Rivi`ere.
N. Potlapally, and L. Torres. Hardware mechanisms for Confidentiality-preserving content-based
memory authentication: A survey of existing publish/subscribe: A survey. ACM Computing Surveys,
techniques and engines. Transactions on Computational 2016.
Science IV, Lecture Notes in Computer Science [21] C. Raiciu and D. S. Rosenblum. Enabling
(LNCS), pages 1–22, March 2009 2009. confidentiality in content-based publish/subscribe
[11] P. Eugster, P. Felber, R. Guerraoui, and A.-M. infrastructures. In Securecomm and Workshops, 2006,
Kermarrec. The many faces of publish/subscribe. ACM pages 1–11. IEEE, 2006.
Computing Surveys, 2003. [22] B. Rogers, S. Chhabra, M. Prvulovic, and Y. Solihin.
[12] S. Gueron. A memory encryption engine suitable for Using address independent seed encryption and bonsai
general purpose processors. Cryptology ePrint Archive, merkle trees to make secure processors os- and
Report 2016/204, 2016. http://eprint.iacr.org/. performance-friendly. In 40th Annual IEEE/ACM
[13] Intel. Intel software guard extensions programming International Symposium on Microarchitecture
reference 329298-002, 2014. https://software.intel.com/ (MICRO 2007), pages 183–196, Dec 2007.
sites/default/files/managed/48/88/329298-002.pdf. [23] F. Schuster, M. Costa, C. Fournet, C. Gkantsidis,
[14] M. Ion, G. Russello, and B. Crispo. Supporting M.Peinado,G.Mainar-Ruiz,andM.Russinovich.VC3:
publication and subscription confidentiality in pub/sub Trustworthy data analytics in the cloud using SGX. In
networks. In Proceedings of the 6th International ICST 2015 IEEE Symposium on Security and Privacy, pages
Conference on Security and Privacy in Communication 38–54, May 2015.
Networks (SecureComm 2010),pages272–289.Springer, [24] M. A. Tariq, B. Koldehofe, A. Altaweel, and
2010. K. Rothermel. Providing basic security mechanisms in
[15] S. Kim, Y. Shin, J. Ha, T. Kim, and D. Han. A first broker-less publish/subscribe systems. In Proceedings of
step towards leveraging commodity trusted execution the Fourth ACM International Conference on
environments for network applications. In Proceedings Distributed Event-Based Systems, DEBS ’10, pages
of the 14th ACM Workshop on Hot Topics in Networks, 38–49, New York, NY, USA, 2010. ACM.
HotNets-XIV, pages 7:1–7:7, New York, NY, USA, [25] A. V. Uzunov. A survey of security solutions for
2015. ACM. distributed publish/subscribe systems. Computers &
[16] G. Klein, K. Elphinstone, G. Heiser, J. Andronick, Security, 2016.
D. Cock, P. Derrin, D. Elkaduwe, K. Engelhardt, [26] A. Wun and H.-A. Jacobsen. A policy management
R. Kolanski, M. Norrish, T. Sewell, H. Tuch, and framework for content-based publish/subscribe
S. Winwood. sel4: Formal verification of an os kernel. middleware. In R. Cerqueira and R. H. Campbell,
In Proceedings of the ACM SIGOPS 22Nd Symposium editors, Proceedings of Middleware 2007:
on Operating Systems Principles, SOSP ’09, pages ACM/IFIP/USENIX 8th International Middleware
207–220, New York, NY, USA, 2009. ACM. Conference, pages 368–388, Berlin, Heidelberg, 2007.
[17] G. Li, S. Hou, and H.-A. Jacobsen. A unified approach Springer Berlin Heidelberg.
to routing, covering and merging in publish/subscribe [27] A.C.-C.Yao.Howtogenerateandexchangesecrets.In
systems based on modified binary decision diagrams. In Proceedings of the 27th Annual Symposium on
Proceedings of the 25th IEEE International Conference Foundations of Computer Science, SFCS ’86, pages
on Distributed Computing Systems, ICDCS ’05, pages 162–167, Washington, DC, USA, 1986. IEEE Computer
447–457, Washington, DC, USA, 2005. IEEE Computer Society.
Society. [28] K. Zetter. NSA Hacker Chief Explains How to Keep
[18] J. Li, C. Lu, and W. Shi. An efficient scheme for Him Out of Your System. Wired, Jan. 2016.
preserving confidentiality in content-based [29] Y. Zhao and D. C. Sturman. Dynamic access control in
publish-subscribe systems. Technical Report a content-based publish/subscribe system with delivery
GIT-CC-04-01, Georgia Institute of Technology, 2004. guarantees. In Proceedings of the 26th IEEE
[19] M. Nabeel, N. Shang, and E. Bertino. Efficient privacy International Conference on Distributed Computing
preserving content based publish subscribe systems. In Systems, ICDCS ’06, Washington, DC, USA, 2006.
Proceedings of the 17th ACM Symposium on Access IEEE Computer Society.
Control Models and Technologies, SACMAT ’12, pages
133–144, New York, NY, USA, 2012. ACM.