Table Of Content

This paper will appear in the proceedings of the 1996 ACSAC Conference. A Case Study of Two NRL Pump Prototypes Myong H. Kang Ira S. Moskowitz Bruce E. Montrose James J. Parsonese Information Technology Division, Mail Code 5540 Center for High Assurance Computer Systems Naval Research Laboratory, Washington, D.C. 20375 Abstract for a (cid:12)nal, production level Pump. The (cid:12)rst exposition of the Pump [4, 5] was designed As computersystems become moreopen and intercon- forone,andonlyone,user/process(Low),sendingmes- nected, theneed forreliableandsecure communication sages to one, and only one, user/process (High) at a also increases. The NRL Pumpwas introduced to bal- higher security level. This problem grew out of the ance therequirements ofreliability,congestioncontrol, need for secure and reliable data replication, see [3]. fairness,andgoodperformanceagainstthoseofthreats When we wish to be speci(cid:12)c, we will refer to this as fromcovert channels and denial of service attacks. the \basic Pump." The E-Pumpis an implementation Inthispaper,wedescribetwoprototypee(cid:11)orts. One ofthebasicPump. Toensure security onemustnotal- implements the Pump at the process (top) layer in lowany back-tra(cid:14)c fromHigh to Low. Unfortunately, terms of a 4-layer network reference model and the this prevents High from sending back to Low an ac- otherimplementsthePumpatthetransport layer. We knowledgement(ACK)thatamessagewassuccessfully then discuss lessons learned and how these lessons will received. ACKs are necessary to make Low-to-High beused indecidinguponthe(cid:12)nalhardwareimplemen- communication both reliable and recoverable. Com- tation of the Pump. munication without ACKs is unreliable because Low doesnotknowifthemessagearrivesatHigh. Commu- nication without ACKs is unrecoverable because Low 1 Introduction mayremove the message before High actually receives In 1993,Kang andMoskowitz(cid:12)rst developed the NRL it. The security problem with ACKs is that the tim- Pump. Since then the Pump theory has been re(cid:12)ned ing of the ACKs can be used to covertly send infor- and extended in a series of theoretical papers [4, 5, mation from High to Low. For example, if High can 6] and applications [8, 2]. This paper discusses the force the ACKs to arrive at Low at either 1 or 2 ms, various issues that arose when two implementationsof with the choice made by High, then this communica- the Pump were developed. These are referred to as tion is a covert channel which has a capacity [7, 9] of the Event Driven Pump (E-Pump), implemented by log2 1+2p5 (cid:25) :69 bits per ms. An obvious solution to Montrose[8],andaversionofthePumprunningonthe this problem would be to remove High’s ability to in- DOSoperatingsystem,implementedbyParsonese,(D- terfere withthe timingofACKs toLow. However, this Pump) [11]. The D-Pump and the E-Pump are based solutionimpedes performance because one wouldhave onverydi(cid:11)erentphilosophies,andbothhavetheirpros to adopt a worse-case approach and send the ACKs at andcons. Wewillanalyzethe design decisionsand the (cid:12)xed time intervals. performance trade-o(cid:11)s between the two philosophies, Kang, Moskowitz, and Lee [6] have extended the andwillmakeuseofthelessonslearnedinourdecisions Pump to the network environment to handle the situ- 1 Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 3. DATES COVERED 1996 2. REPORT TYPE 00-00-1996 to 00-00-1996 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER A Case Study of Two NRL Pump Prototypes 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION Naval Research Laboratory,Code 5540,4555 Overlook Avenue, REPORT NUMBER SW,Washington,DC,20375 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 11. SPONSOR/MONITOR’S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 13. SUPPLEMENTARY NOTES 14. ABSTRACT 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF ABSTRACT OF PAGES RESPONSIBLE PERSON a. REPORT b. ABSTRACT c. THIS PAGE 12 unclassified unclassified unclassified Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18 ation of multiple (non-communicating amongst them- (Stochastic behavior) The Pump uses a secure (in- selves) users/processes at the low level (Lowi) sending direct) High to Low (cid:13)ow control mecha- a message to one of the multiple (non-communicating nism. The Pump keeps track of the rate amongst themselves) users/processes at a (cid:12)xed higher at which High takes messages out of the level (Highj). When we wish to be speci(cid:12)c we will Pumpandsends ACKs toLowbased upon refer to this as the \network Pump." The network a movingaverage of the High rate. We ac- Pump deals with the added element of fair allocation complishthisbyhavingtheLowACKtime ofresources. The D-Pumpis animplementationofthe be a draw of a randomvariable with mean network Pump. equal to the High moving average (MA). Whenwediscusssomethingthatappliestoeitherthe Thus,byusingastochastic approachtodi- D-PumportheE-Pump,wewillusetheterm\Pump," lutetheHighin(cid:13)uenceonthePump’sACK and the terms \Low" and \High" will refer to client to Low, we minimize the covert channel applications at di(cid:11)erent security levels. threat. However, the average behavior still Pump re(cid:13)ects High’s long term behavior, which ensuresgoodperformance. Infactthebasic n Low messages . . . MA messages High Pumpperformsas wellas the (non-secure) ACK ACK store and forwardprotocol (SAFP) [5]and buffer the network Pump performs as well as any other protocol under round-robin schedul- Figure 1. The simpli(cid:12)ed Pump architecture ing [6]. The Pump (see (cid:12)gure 1) has several key ideas as follows: (Intermediatebu(cid:11)er) The Pump, for security rea- The timelineof ACKs are shown in (cid:12)gure 2. sonsmentionedpreviously,disconnects any Low Pump Pump High directcommunicationfromHightoLowby actingasanintermediarybetweenLowand e High. Low sends its message to the Pump, tim message a not to High. The Pump receives the mes- sy sagefromLowandretainsitinstablebu(cid:11)er stoticmhaestic nch message r until it is accepted by High. The Pump, o n then sends an ACK to Low, thus replac- A c k ou ingdirectcommunicationbetweenLowand s A c k High by a mediated indirect communica- tionwhile preserving reliability,and recoverability. Furthermore, the Pumpuses his- torical knowledge of past High behavior to Figure 2. Time diagram of message passing ensure good performance. from Low to High (Handshake protocol) The Pumpuses ahandshake protocoltoguaranteereliability. Lowwaits Note that we are not being speci(cid:12)c about what a tosend anew messageuntilithas received \message" is. This is because it depends on which anACKofitslastmessage,fromthePump. Pump implementationis used. The E-Pump uses \ac- The Pump can also be used with a slid- tual" messages and their ACKs, whereas the D-Pump ing window scheme that will allow a cer- uses a TCP packet as a message and modulates the tain number of un-ACKed messages to be TCP ACKs. We will explain this in detail later on in outstanding. the paper. 2 4-layer model connecting two sees the packet. Every Ethernet packet has a header that includes the source and destination Ethernet ad- systems dress. Each machine is designed to pay attention only topacketswithitsownEthernetaddressinthedestina- Werestrict ourattentiontothefollowing(see (cid:12)gure3) 1 tion(cid:12)eld. Mostnetworksuse Ethernet. Unfortunately, network architecture (similar to [10]) to explain dif- Ethernethasitsownaddresses. ThedesignersofEther- ferent Pump implementations. net wanted to make sure that no two machines would 1. Process layer | P.L. This is the top-most have the same Ethernet address. Furthermore, they layer, and the layer that application processes didnotwanttheuser tohavetoworryaboutassigning (e.g,. FTP, mail,etc.) use for communication. Ethernet addresses ( incomparisontoassigningIPad- dresses). So each Ethernet card comes with a unique 2. TCP layer|T.L.Thecommunicationprotocol Ethernet address built-infromthe factory. calledtransmissioncontrolprotocol(TCP)isused to providereliableservices to the aboveP.L.TCP reliablysends packets between the sender and the 2.2 TCP/IP receiver. Our concern is computers that use internet protocol (IP) to communicate. Each individualcomputer has a 3. IP layer | I.L. The internet protocol (IP) pro- unique IP address (e.g., 192.6.16.23). The IP network vides the basic packet delivery service to TCP. software generally needs a 32-bit IP address in order 3 4. Ethernet layer | E.L. This layer handles the toopen aconnection or send a datagram . The IP ad- protocols needed to manage a speci(cid:12)c physical dress has two parts: the network address and the host medium,such as Ethernet. address. If source and destination have the same network addresses, we say they are on the same network; Process layer (P.L) Process layer otherwise, they are on a di(cid:11)erent network. IP assumes that a system is attached to some local TCP layer (T.L.) TCP layer network. It assumes that the system can send data- gramstoanyother systemonthe samenetwork. Since IP layer (I.P.) IP layer physical we are assuming that Ethernet is handling the physi- connection cal layer communication,IP simply(cid:12)nds the Ethernet Ethernet layer (E.L.) Ethernet layer address of the destination system, and puts the datagram (packet) out on the Ethernet. If the destination Figure 3. Reference model of network layers is on a di(cid:11)erent network, the datagram is sent to a 4 gateway using a routing protocol. When the gateway BoththeE-PumpandtheD-PumpuseTCPtosend receives the datagram,itwillforward the datagramto packets. An important distinction between the two the destination that is on a di(cid:11)erent network than the Pumpimplementationsis that the E-Pumpmodulates originatingnetwork. P.L. ACKs (the true process ACK), whereas the D- Note that there is no relationship between the Eth- Pump modulates T.L.ACKs (TCP ACKs). ernet address and the IP address. There is a separate protocol for associating an Ethernet address to an IP 2.1 Ethernet address, called \address resolution protocol (ARP)." An Ethernet cable can be used for the communication That association is stored in the ARP table. Suppose between Ethernet cards ondi(cid:11)erent computers. Ether- a process on system 192.6.16.9 (class C IP address) net is a \broadcast medium." When a packet2 is sent wants to connect to a process on system 192.6.16.7. out on the Ethernet, every machine on the network The originatingsystem will (cid:12)rst verify that 192.6.16.7 1 3 Ourdescriptionisnotastandardnetworkdescription.How- DatagramisaunitofmessageattheIPorTCP layers. 4 ever, it is the description most relevant to understanding the A gateway(or router)is a systemthatconnectsa network currentprototypesofthePump. with one or more other networks. Gateways are often normal 2 PacketisaunitofmessageattheEthernetlayer. computersthathappentohavemorethanonenetworkinterface. isonthesamenetwork,soitcantalkdirectlyviaEther- 3 The E-Pump net. Thenitwilllookup192.6.16.7initsARPtable,to see ifitalreadyknowstheEthernet address. Ifthesys- The E-Pump is software that resides on Wang’s temis not in the ARP table, it uses the ARP protocol (HFSI’s) XTS-300 platform. The XTS-300 was cho- to broadcast an ARP request. When the destination sen because of (1) its availability,and (2) its B3 rated system sees an ARP request for itself, it is required to operatingsystem(STOP4.1). Weuseanevaluatedop- respond. So 192.6.16.7 will see the request, and will erating system in order to avoidevaluatingthe system respond withanARPreplysayingine(cid:11)ect \192.6.16.7 services that are used by the E-Pump (e.g., interpro- (IPaddress)is8:0:20:1:55:33(Ethernetaddress)." Now cess communication, (cid:12)le-system, etc.). STOP can be the communicationcan proceed. describedbyringsofprivilegesasshownin(cid:12)gure4and discussed below. TCP, which handles communication at the T.L., is i ng r - responsible for makingsure that messages get through 3 to the other end. It keeps track of what is sent, and r i ng- 2 retransmits anything that did not get through. If any message is too large for one datagram,e.g. the text of r ing-1 agrmamaisl,maensdsamgea,kTeCsPurewitlhlasptlitthietyuapllinatrorisveevecroarlredcattlay-. ware r ing-0 COMMODITY Sincethese functionsareneededformanyapplications, oft APPLIC they are put together into a separate protocol, rather S ATOI tOtnhiennateewnsocbtrahkenainctogtahmppinpmaklriutcnoaoiftfciaoTttnhCisoePncsaspnawescuiit(cid:12)sfhoecraawmntihoionetnnhgsetarfhocerloyibmsnerpnaereudydtinerorge.flmiSraioabmuille--. detsurTTCBSeScyusrtietmy SKee rrnvelices SECIVRESMETSYSN ilarly, TCP calls on the services of IP. Although the sceartvioicness, tthhaerteTaCrePsstuilplpsloimeseakreinndeseodfedapbpylicmaatinoynsaptphlait- Application Soft w a r e donotneed them(i.e.,someapplicationsmayuseUser Datagram Protocol (UDP) instead of TCP). Note [1] has implementedone-way links at the T.L. Figure 4. STOP security ring structure The security kernel provides basic system operating services (e.g., resource management, process schedul- ing, interrupt, trap handling) and enforcement of sys- 2.3 Processes tem security (e.g., security and integrity rules). Privi- legedsoftware knownas Trusted Softwareprovidesad- There are many P.L. protocols (e.g., mail, FTP). The ditionalsecurity services outside the kernel. Commod- applications of interest with respect to the Pump are ity Application System Service (CASS) provides un- thosethatpassmessagesfromonehosttoanother(e.g., trusted operating system services to application pro- data replication, mail, FTP). The job of the Pump is gramson the XTS-300. to act as an intermediary inthe action of Low sending The XTS-300 supports trusted processes. A process a message to High, and mimicking the ACK of this is trusted if the process has privileges that exempt it message from High back down to Low. The E-Pump fromspeci(cid:12)caccess controlrules(e.g.,noread-uporno implements the Pump protocol at the P.L. (i.e., the write-down rule). Since the basic Pump sends ACKs E-Pump modulates P.L. ACKs). In contrast, the D- back to Low, a portion of the basic Pump must have Pump modulates the TCP layer ACKs. Of course, we trusted processes. musthavehigh-assurancethatourimplementationsdo Ideally, the entire E-Pump should be implemented not allow the high side to a(cid:11)ect ACKs at any of the as trusted ring-2 processes. Since ring-2 processes on other layers. the XTS-300 currently cannot access TCP/IP, it was necessary to implement the portions of the Pump re- Output DataFIFO. It also sends quiring access to TCP/IP in ring-3. The (cid:12)le-system is an IPC message to the Moving the only mechanism available for communication be- Average Object to trigger the tween ring-3 to ring-2 processes on the XTS-300. We movingaverage computation. used special(cid:12)les, called FIFOs, witha(cid:12)rst-in-(cid:12)rst-out ACK It computes the randomized de- protocol, as the communication channel between the lay based on the moving average ring-2 and ring-3 components of the E-Pump. and sends an ACK to the Low We refer to each component of the Event Driven ACK FIFO after data is safely Pump as an Object. Each Object was implemented stored in the non-volatileSAFB. as a separate process designed to accomplisha speci(cid:12)c taskwhencertaineventsoccurred. Aneventiseithera Lo Timer It is the timer for the ACK Ob- messagesentbyanotherObjectorthecompletionofan ject. I/O operation. Inter-Object message passing was ac- complishedviatheInterprocess Communication(IPC) Moving Average It computes the moving average interface provided by the operating system. and sends the updated moving Two store and forward bu(cid:11)ers (SAFB) were used; average to the ACK Object. one in volatile shared memory and the other in non- Hi Timer ItisthetimerfortheMovingAv- volatile disk storage. The two bu(cid:11)ers together make erage Object. up the \Pump bu(cid:11)er" for performance and recover- abilityreasons. High reads out ofvolatilememorybut Disk Writer It writes data to the disk and ACKs are not sent to Low until the message is writ- sends an IPC message to the ten to disk. All ring-2 Objects have access to both ACKObjectwhichindicatesthat SAFBs. The non-volatile SAFB was required for re- the data is safely stored in covery purposes in the event the XTS-300 should halt the non-volatile SAFB. Unfortu- duringoperation. ThevolatileSAFBserves astheI/O nately, this is a costly timeoper- bu(cid:11)er for data transferred to the non-volatile SAFB, ation. the Output FIFO, and the Input FIFO. Free Record It reads an ACK from the High A brief description of each Object follows. ACK FIFO, sends an IPC mes- S2F It delivers data from the low ap- sage to the Moving Average Ob- plicationtotheInputDataFIFO ject so that the new moving av- andrelaysthePump’sACKfrom erage can be computed, and re- the Low ACK FIFO to the low moves the data from the disk application. (Zap Hdr). Then it sends an IPC message to the Memory Writer Memory Writer ItreadsdatafromtheInputData Object to indicate the space is FIFO,storesthedataintoShared now available. Memory,andsendsIPCmessages F2S It delivers data from the Output to the Disk Writer and Memory Data FIFO to the high applica- Reader Objects, informing them tionand sends ACKs to the High thatthereisdatatobeprocessed. ACKFIFOuponsuccessfuldeliv- It also sends an IPC message to ery of the data to the high appli- the ACK Object so that it can cation. keep track of timing (e.g., timeout (NAK), disk writing overhead). Memory Reader It reads data from Shared Mem- ory and writes the data to the F2S Hi Ack Timer OuDtaptuat TIMER_ALARM TIMER_ALARM TIMER_START TIMER_STOP TIMER_PEEK TIMER_PEEK HAiCgKh Moving S Average Ack H AR Memory DATA_READYPUMP_READYDATA_SYNC Free D E Reader Record Zap Hdr DM DATA_READY RECORD_AVAILABLE MOV_AVG IS E Memory Disk PUMP_READY K M Writer Writer OR DATA_READY DATA_SYNC Y PUMP_TIMEOUT ACK Ack or Nack RECORD_AVAILABLE IDnaptuat TIMER_ALARM TIMER_ALARM TIMER_START TIMER_STOP TIMER_PEEK TIMER_PEEK LAoCwK TiLmoer Ack or Nack Legend IPC msg FIFO msg Data msg S2F Process Object FIFO Object SAFB Figure 5: The Event Driven Pump Not shown in (cid:12)gure 5 are two additional processes the P.L. called the Recovery Object and the Create Pump Ob- DOS was chosen as a prototyping environment for ject. The Recovery Object is responsible for the or- the D-Pumpinorder to host a TCP/IP protocol stack derly recovery from system failure. The Create Pump that could easily be ported to any other host or em- Object sets up all data structures and creates all the bedded system. Furthermore, the use of DOS, which other Objects described above including the Recovery isone ofthe simplestoperatingsystems, facilitatesthe Object. Unlike the other Objects described above, the developmentofcustomhardware thatwould providea Recovery Object and Create Pump Object will termi- high-level of assurance that the D-Pump code reliably nate after their tasks have been completed. performs the functionalityof the (network) Pump. TheE-Pumpuses acustomizedPumpprotocol(e.g., The followingsteps describe the D-Pump: [8]) which is a P.L. protocol, in contrast, the D-Pump 1. The Pump listens to ARP requests. uses TCP protocol to communicate. The E-Pump modulatesP.L.ACKs according to the basic Pumpal- 2. If an ARP request contains an IP address of any gorithm[8] as shown in (cid:12)gure 6. high systems to which the Pump is con(cid:12)gured to E-Pump deliver messages, then the Pump replies to the ARP request by sending its lowEthernet address. buffer 3. The lowsender establishes the TCPconnection to P.L. P.L. P.L. P.L. the Pump. T.L. T.L. T.L. T.L. 4. The Pump establishes the TCP connection to the I.L. I.L. I.L. I.L. high receiver on behalf of the low sender (the E.L. E.L. E.L. E.L. Pump uses IP addresses and port numbers that 5 are availableto the Pump ). Figure 6: Communicationpath of E-Pump 5. Activities at the low side of the Pump: (a) The low portion of the Pump receives mes- 4 The D-Pump sages and puts them in the Pump’s bu(cid:11)er. (b) Send TCPACKs tothe lowsender according TCPisusedbymanyapplications(e.g.,FTP)toensure to the Pump algorithm. reliabilityat the T.L. Due to this we thought that the Pump could be implemented at the T.L. by means of (c) Continuestep(a)and(b)untilthelowsender adjustingthe TCP ACK stream. This implementation discontinues the connection. ofthePumpiscalledtheD-Pump. Thismodi(cid:12)edTCP 6. Activities at the high side of the Pump: should be transparent to the application. In section 5 we discuss problems that arise fromimplementingthe (a) The Pump delivers messages from its bu(cid:11)er Pump at the T.L. to the high receiver. The D-Pump exists as software on a single board (b) The Pump receives ACKs from the high re- 486class PC with one lowEthernet card and one high ceiver and updates the movingaverage. Ethernet card. It implementsthe network Pump algo- (c) Continuestep(a)and(b)untilthelowsender rithm[6]. This algorithmis amodi(cid:12)cationofthe basic discontinues its connection and there are no Pumpalgorithmdesigned todeal with the added issue more messages destined for the high receiver of multiplelow senders at the same security level, and in the bu(cid:11)er. multiplehigh receivers at the same security level. It is further assumedthatthe Lows(Highs) donotcommu- 7. The lowsender discontinues the connection tothe nicate among themselves. The modi(cid:12)cation does not Pump. a(cid:11)ectourcomparisonsinthispaper. TheD-Pumpuses 5 Wemodi(cid:12)edtheIPlayersothataddressinformationisavail- customizedTCPsoftwarethatmodulatesthe ACKs in able to the T.L. We also modi(cid:12)edthe Ethernet layer to time- the T.L. Recall that the E-Pump modulates ACKs in stamppackets. 8. ThePumpdiscontinuestheconnectiontothehigh Hence, theD-Pumpdoesnotneedanycustomized receiver. address scheme because it uses the TCP protocol. D-Pump However,since the E-Pumpcannotaccess thisad- dress informationat the P.L., the address scheme buffer P.L. P.L. mustbeaddedtothePumpprotocol,thusincreas- ing overhead. T.L. T.L. T.L. T.L. I.L. I.L. I.L. I.L. The above reasons seem to imply that implement- E.L. E.L. E.L. E.L. ing the Pump at the T.L. is a good design decision. However, as we will discuss in the following sections, Figure 7: Communication path of D-Pump problems exist with our T.L. implementation. 5.2 The Pump and its wrappers 5 Comparison of the E-Pump and the D-Pump The Pump is designed to be an application indepen- dentdevice, receiving messagesfromLow(sender) and In this section, we compare the two approaches of our deliveringthemtoHigh(receiver). We emphasizethat prototypes. The major di(cid:11)erence of interest to us in the Low to High communicationis actually a commu- this paper is that the E-Pump implements the Pump nicationbetweenapplicationprogramsrunningonLow at the P.L., while the D-Pump implements the Pump and High computers. Also, note that application pro- at the T.L. grams that expect some data values coming back as a result of computation cannot be used with the Pump 5.1 Performance because the Pump does not allow data values to pass through itself. Therefore, the only application pro- Withrespect tooverhead,the D-Pumpisamuchmore grams that can utilize the Pump are those which can e(cid:14)cient implementation than the E-Pump. Some of function with only simple ACKs (or no ACKs) from the additional overhead of the E-Pumpis as follows: the receiver program. (cid:15) Since ring-2 processes on the XTS-300 cannot ac- In general, an application program that functions with (simple) ACKs is able to ensure reliability at cess TCP/IP, the portions of the E-Pump that the application level. Thus, when the sender program are needed to access TCP/IP were implemented sends amessage,it expects the receiver programtore- in ring-3. Hence, four extra FIFOs (Output data, turn an ACK in the speci(cid:12)c formatdetermined by the Input data,HighACK, andLowACKin(cid:12)gure 5) application protocol. If the sender program directly and two extra processes (F2S and S2F) are used sends a message to the Pump, the Pump, which is an inthe E-Pumpwhichareunnecessary inotherim- applicationindependentdevice, cannotreturn anACK plementations. in the formatspeci(cid:12)ed by the application protocol. (cid:15) The E-Pump writes every message from Low to Therefore, wrappers, which are application-speci(cid:12)c, disk,forrecovery reasons,beforeitsendsanACK. are needed to ensure the correct formatting of appli- However, if the D-Pump wrote every message to cation ACKs. In other words, wrappers satisfy both disk(whichisthemostdesirablerecoverabilityop- the application-speci(cid:12)c protocol and the Pump proto- tion) then the same additionaloverhead would be col. Even thoughthe D-Pumpcan satisfythe ACKre- incurred. quirementattheT.L.,itcannotsatisfytheapplication- (cid:15) The messagedoes nothave togouptothe P.L.in speci(cid:12)c ACK requirements at the application layer. Hence, both Pumps require wrappers to correctly in- theD-Pump(i.e.,itstopsatT.L.),thusitrequires terface with the speci(cid:12)c application program. We em- less processing time. phasize that for most applications that utilize TCP, (cid:15) SincetheD-Pump’strustedcodeincludesTCP/IP satisfying TCP’s requirements are not enough to sat- code,itcanaccess IPaddresses andportnumbers. isfy the application-speci(cid:12)c protocol requirements. (cid:15) The high wrapper receives a Pump message. application application independent application specific device specific If more Pump messages are required to con- struct a complete application message, then a p pp lrlooicwgartaimon 1 w r laopwper 2 Pump (1) w hraigphper (2) a p pp rhloiicggahrtaiomn go to (4). 4 3 (4) (3) (2) The high wrapper sends anapplicationmessageto play a role as a play a role as a the high applicationprogram. high application low application (3) The high wrapper receives an ACK from the high application program. Figure 8. The Pump and wrappers (4) ThehighwrappersendsanACKtothePump. The 6 A brief description of wrappers and their interactions Pump message is now removed from the Pump’s withthePumpandapplicationprograms,asillustrated bu(cid:11)er upon receiving the ACK. Go to (1). in (cid:12)gure 8, is as follows: Aswediscussed earlier,ingeneral,bothPumpsneed 1. The low application program sends (if available) application-speci(cid:12)cwrappersattheP.L.Themajordif- an application message to the low wrapper. The ferences between the two Pump implementationsare: low wrapper reformats the application message into the format that the Pump understands, and (cid:15) Since the E-Pump has been implemented at the produces a Pump message(s). Sometimesone ap- P.L., the lower layer protocols are transparent to plicationmessagemaybe transformedintoseveral the E-Pump. Pump messages. Note that in the E-Pump, mes- (cid:15) Since the D-Pump has been implemented at the sages are created at the P.L.and ACKs are in the T.L.,TCP requirements must also be satis(cid:12)ed. P.L.,whereasintheD-Pump,messagesandACKs are at the T.L. 5.3 Reliability & Recoverability 2. The low wrapper then sends a Pump message to We wish to have reliable communication between the the Pump. sender and the receiver. The Pump should guarantee 3. ThelowwrapperreceivesanACKfromthePump. reliabilityfrom the sender to the Pump, and fromthe Pump to the receiver. However, reliabilityis not com- (cid:15) Repeat 2 and 3 until the Pump message(s) posable. For example, after the Pump receives a mes- corresponding to the original application sage and the sender receives an ACK, the sender may message have been passed to and ACKed by perform garbage collection. If, at this time, the com- the Pump. municationfrom the Pump to the receiver breaks and later it is re-established, then orderly recovery from 4. The low wrapper sends an application speci(cid:12)c system failure between the Pump and the receiver is ACK to the sender program. Go to 1. 7 needed to guarantee reliable and secure sender to receiver communication. Hence, the low wrapper is a proxy of the high appli- The D-Pumpdoes not wait for ACKs fromthe high cation program that receives a message from the low application (or wrapper if the use of wrappers is re- applicationprogram. quired) but rather it uses TCP ACKs for computing Similarly, the high wrapper is a proxy of the low the movingaverageandgarbage collection. Ingeneral, application program that receives a message from the TCP ACKs from High to the Pump are controllable Pump and deliver it to the high application program. neither by the application program nor by the wrappers. Hence, there is a possibility the packet will be (1) The Pump sends (if available)a Pump message to 7 the high wrapper. If the sender and receiver applications are recoverable (e.g., database replication) then the Pump also preserves the 6 Adetailedexampleofwrapperscanbefoundin[2]. recoverability.