Table Of ContentDistributed-Memory Computing With
the Langley Aerothermodynamic
Upwind Relaxation Algorithm (LAURA)
(cid:3) y
Christopher J. Riley and F. McNeil Cheatwood
NASA Langley Research Center, Hampton, VA 23681
Paper Presented at the
4th NASA National Symposium on Large-Scale Analysis and Design on
High-Performance Computers and Workstations
Oct. 15{17, 1997/Williamsburg, VA
TheLangleyAerothermodynamicUpwindRelaxationAlgorithm(LAURA),aNavier-
Stokes solver, has been modi(cid:12)ed for use in a parallel, distributed-memory environment
usingthe Message-Passing Interface(MPI) standard. A standarddomaindecomposition
strategy is used in which the computational domain is divided into subdomains with
each subdomain assigned to a processor. Performance is examined on dedicated parallel
machines and a network of desktop workstations. The e(cid:11)ect of domain decomposition
andfrequencyofboundaryupdatesonperformanceandconvergenceisalsoexaminedfor
several realistic con(cid:12)gurations and conditions typical of large-scale computational (cid:13)uid
dynamicanalysis.
Introduction though the LAURA computer code is continually be-
The design of an aerospace vehicle for space trans- ingupdated withnewcapabilities, itisamaturepiece
portation and exploration requires knowledge of the of software with numerous options and utilities that
aerodynamic forces and heating along its trajectory. allow the user to tailor the code to a particular appli-
13
Experiments(bothground-testand(cid:13)ight)andcompu- cation.
tational (cid:13)uid dynamic (CFD) solutions are currently LAURAwasoriginallydevelopedandtunedformul-
used to provide this information. At high-altitude, tiprocessor, vector computers with shared memory
high-velocity conditions that are characteristic of at- such as the CRAY C-90. Parallelismusing LAURA is
mospheric reentry, CFD contributes signi(cid:12)cantly to achievedthrough the use of macrotaskingwhere large
the design because of the ability to duplicate (cid:13)ight sections of code are executed in parallel on multiple
conditionsandtomodelhightemperaturee(cid:11)ects. Un- processors. Because LAURA employs a point-implicit
fortunately, CFD solutions of the hypersonic, viscous, relaxation strategy that is free to use the latest avail-
reacting-gas(cid:13)owoveracompletevehiclearebothCPU able data from neighboring cells, the solution may
time and memory intensive even on the most power- evolve without the need to synchronize tasks. This
ful supercomputers; hence, the design role of CFD is resultsin averye(cid:14)cientuseofthe multitaskingcapa-
14
generally limited to a few solutions along a vehicle’s bilities of the supercomputer. But future supercom-
trajectory. puting may be performed on clusters of less powerful
One CFD code that has been used extensively for machines that o(cid:11)er a better price per performance
the computation of hypersonic, viscous, reacting-gas than current large-scalevector systems. Parallelcom-
(cid:13)owsoverreentryvehiclesistheLangleyAerothermo- puterssuchastheIBMSP2consistoflargenumbersof
1,2
dynamic Upwind Relaxation Algorithm (LAURA). workstation-classprocessorswith memory distributed
LAURAhasbeenusedinthepasttoprovideaerother- among the processors instead of being shared. In
modynamic characteristics for a number of aerospace addition, improvements in workstation processor and
3 4 5
vehicles (e.g. AFE, HL-20, Shuttle Orbiter, Mars network speed and the availability of message-passing
6 7
Path(cid:12)nder, SSTO Accessto Space ) and is currently librariesallow networks of desktop workstations(that
being used in the design and evaluation of blunt aero- may sit idle during non-work hours) to be used for
15
braking con(cid:12)gurations used in planetary exploration practical parallel computations. As a result, many
8,9
missions and Reusable Launch Vehicle (RLV) con- CFD codes are making the transition from serial to
10,11 12 16{20
cepts (e.g. the X-33 and X-34 programs). Al- parallel computing. The current shared-memory,
macrotasking version of LAURA requires modi(cid:12)ca-
(cid:3)
Research Engineer, Aerothermodynamics Branch, Aero- tion before exploiting these distributed-memory par-
andGas-DynamicsDivision.
y allel computers and workstation clusters.
ResearchEngineer,VehicleAnalysisBranch,SpaceSystems
andConceptsDivision. Several issues need to be addressed in creating a
1of9
distributed-memory version of LAURA: 1) There is
thechoiceofprogrammingparadigmtouse. Adomain
17
decompositionstrategy (which involvesdividing the
computationaldomain into subdomainsand assigning
eachtoaprocessor)isapopularapproachtomassively
parallel processing and is chosen due to its similarity
to the current macrotasking version. 2) To mini- PARTITIONS
mize memory requirements, the current data struc-
ture of the macrotasking, shared-memory version is
changed since each processorrequires storageonly for
its own subdomain. 3) The choice of message-passing
library(whichprocessorsusetoexplicitlyexchangein-
formation) may impact portability and performance.
4) The frequency of boundary data exchanges be-
tween computational subdomains can in(cid:13)uence (and
BLOCKS
may impede) convergence of a solution although the
point-implicit nature of LAURA already allows asyn-
14
chronous relaxation. 5) There are also portabil-
ity and performance concerns involved in designing a Fig. 1 Domain decomposition of macrotasking
version of LAURA to run on di(cid:11)erent (cache-based version.
and vector) architectures. 6) Finally, a distributed-
relaxation equation is
memory, message-passing version of LAURA should
retain all of the functionality, capabilities, utilities, ML(cid:14)qL =rL (1)
and easeof useof the currentshared-memoryversion.
where ML is the n x n point-implicit Jacobian, qL
This paper describes the modi(cid:12)cations to LAURA
is the vector of conserved variables, rL is the resid-
that permit its use in a parallel, distributed-memory
ualvector,andn isthe number of unknown variables.
environment using the Message-Passing Interface
21 For a perfect gas and equilibrium air, n is equal to 5.
(MPI) standard. An earlier, elementary version of
For nonequilibrium chemistry, n is equal to 4 plus the
LAURA for perfect gas (cid:13)ows using the Parallel Vir-
22 number of constituent species. The residual vector rL
tualMachine(PVM)library providesaguideforthe
23 and the Jacobian ML are evaluated using the latest
current modi(cid:12)cations. Performance of the modi(cid:12)ed
availabledata. Thechangeinconservedvariables,qL,
version of LAURA is examined on dedicated paral-
maybecalculatedusingGaussianelimination. AnLU
lel machines (e.g. IBM SP2, SGI Origin 2000, SGI
factorizationoftheJacobiancanbesaved(frozen)over
multiprocessor) as well as on a network of worksta-
large blocks of iterations ((cid:25) 10 to 50) to reduce com-
tions (e.g. SGI R10000). Also, the e(cid:11)ect of domain
putational costs as the solution converges. However,
decomposition and frequency of boundary updates on
the Jacobian will need to be updated every iteration
performance and convergence is examined for several
earlyinthecomputationwhenthesolutionischanging
realisticcon(cid:12)gurationsandconditionstypicaloflarge-
rapidly.
scale CFD analysis.
Macrotasking
LAURAutilizesmacrotaskingbyassigningpiecesof
LAURA
the computational domain to individual tasks. First,
LAURA is a (cid:12)nite-volume, shock-capturing algo- the computational domain is divided into blocks,
rithm for the steady-state solution of inviscid or vis- where a block is de(cid:12)ned as a rectangularly ordered
cous,hypersonic(cid:13)owsonrectangularlyordered,struc- array of cells containing all or part of the solution
tured grids. The upwind-biased inviscid (cid:13)ux is con- domain. Then each block may be subdivided in the
24
structed using Roe’s (cid:13)ux-di(cid:11)erence splitting and computational sweep direction into one or more par-
25
Harten’s entropy (cid:12)x with second-order corrections titions. Partitions are then separately assigned to a
based on Yee’s symmetric total-variation-diminishing task (processor). Figure 1 shows a two-dimensional
26
scheme (TVD). Gas chemistry options include per- (2D) domaindivided into2blocks with eachblockdi-
fectgas,equilibriumair,and airinchemical andther- vided into 2 partitions. Thus a task may work on one
malnonequilibrium. Moredetailsofthealgorithmcan ormore partitionswhich may be containedin asingle
be found in Refs. 1, 2 and 13. block or may overlap several blocks. Each task then
The point-implicit relaxation strategy is obtained gathers and distributes its data to a master copy of
by treating the variables at the local cell center L at the solution which resides in shared memory. With
theadvancediterationlevelandusingthelatestavail- the point-implicit relaxation, there is no need to syn-
able data from neighboring cells. Thus, the governing chronizetaskswhich results in a very e(cid:14)cient parallel
2of9
in terms of computational speed and convergence.
BLOCK Measuring the elapsed wall clock time of the code on
di(cid:11)erent machines estimates the communication over-
COMMUNICATION head and message-passing e(cid:14)ciency of the code. The
BETWEENBLOCKS
communication overhead associated with exchanging
boundary data information between nodes depends
on the parallel machine, the size of the problem, and
the frequency of exchanges. The frequency of data
BOUNDARYDATA
STORAGE exchangesmaybedecreasedifnecessarytoreducethe
communication penalty, but this may adversely a(cid:11)ect
convergence. Therefore, the impact of boundary data
exchange frequency on convergence is determined for
several realistic vehicles and (cid:13)ow conditions.
Computational Speed
Timingestimatesusingthe message-passingversion
of LAURA are presented for an IBM SP2, an SGI
Origin 2000, an SGI multiprocessor machine, and a
networkofSGIR10000workstations. Thesingle-node
Fig. 2 Domain decomposition of message-passing
version. performanceofLAURA ona cache-based(asopposed
to vector) architectureis not addressed. Viscous, per-
implementation. fect gas computations are performed on the forebody
10,11
Message-passing ofanX-33 con(cid:12)gurationwithagridsizeof64x56
x64. Thecomputationaldomainissplitalongeachof
In the new message-passingversion of LAURA, the
thecoordinatedirections(dependingonthenumberof
computationaldomainisagainsubdivided into blocks
nodes) into blocks of equal size. The individual block
along any of the three (i;j;k) coordinate directions
sizesareshownbelow. Becausetheblocksareequalin
with each block assigned to a processor. As com-
pared to the macrotasking version, this is analogous Table 1 Block sizes for timing study.
to de(cid:12)ning each block to contain only one partition
Nodes Block
and assigning each partition to a separate task. The
2 32 x 56 x 64
number of blocks is therefore equal to the total num-
4 32 x 28 x 64
ber of processors. Due to the distributed memory of
8 32 x 28 x 32
theprocessors,eachtaskonlyrequiresstorageonlyfor
16 16 x 28 x 32
its own block plus storage for boundary data from as
32 16 x 14 x 32
manyassixneighboringblocks(i.e. oneforeachofthe
64 16 x 14 x 16
six block faces). Figure 2 shows a 2D domain divided
128 8 x 14 x 16
equally into 4 separate blocks. Each processor works
only on its own block and pauses at user-speci(cid:12)ed in-
tervals to exchangeboundarydata with its neighbors. size, boundary data exchanges are synchronized at a
Theboundarydataexchangeisexplicitlyhandledwith
speci(cid:12)ed iteration interval for convenience. Each run
send and receive calls from the MPI message-passing
begins with a partially converged solution and is run
21
library. The MPI library was chosen because it is
for 200 iterations with second order accuracy. Two
a standard and because there are multiple implemen- values (1 and 20) are used for nexch, the number of
tations that run on workstations as well as dedicated iterations between boundary data exchanges, to esti-
27,28
parallel machines. Synchronization of tasks oc-
mate the communication overhead on each machine.
curs when messages are exchanged, but this exchange
The number of iterations that the Jacobian is held
is not required for any particular iteration due to the (cid:12)xed, njcobian, is equal to 20 and represents a typical
point-implicit relaxation scheme. As in the macro-
value for solutions that are partially converged.
tasking version, tasks (or blocks) of various sizes may
Four di(cid:11)erent architectures are used to obtain tim-
accumulate di(cid:11)ering numbers of iterations during a
ingestimates. The(cid:12)rstisa160-nodeIBMSP2located
run. For blocks of equal size, it may be convenient
attheNumericalAerospaceSimulation(NAS)Facility
to synchronize the messageexchange at speci(cid:12)ed iter-
at NASA Ames using IBM’s implementation of MPI.
ation intervals.
Thesecondisa64processor(R10000)SGIOrigin2000
also located at NAS using SGI’s version of MPI. The
Results third is a 12 processor (R10000) SGI machine oper-
The performance of the distributed-memory, ating in a multiuser environment, and the fourth is
message-passing version of LAURA is examined a network of SGI R10000 workstations connected by
3of9
nexch=1 nexch=1
104 nexch=20 104 nexch=20
linearspeedup linearspeedup
) IBMSP2 ) SGIR10000
sec 103 X-33forebody sec multiprocessor
( (
e ViscousPG e
m m
ti 64x56x64 ti 103
ck 200iterations ck
o o
allcl 102 allcl XV-is3c3ofuosrePbGody
w CRAYC-90 w 64x56x64
(9CPU’s)
200iterations
101 102
100 101 102 1 2 3 4 5 6 7 8 910
nodes nodes
Fig. 3 Elapsed wall clock time on IBM SP2. Fig. 5 Elapsed wall clock time on SGI multipro-
cessor.
nexch=1
104
nexch=20
linearspeedup 104 nexch=1
nexch=20
linearspeedup
SGIOrigin2000
)
c
se 103 ) SGIR10000
me( X-33forebody sec
ti ViscousPG e(
k m
oc 64x56x64 ti 103
wallcl 102 200iterations clock X-33forebody
all ViscousPG
w
64x56x64
200iterations
101
100 101 102
nodes 1021 2 3 4 5 6 7 8 910
nodes
Fig.4 ElapsedwallclocktimeonSGIOrigin2000.
Fig. 6 Elapsed wall clock time on network of SGI
Ethernet. BothoftheseSGImachinesusetheMPICH R10000 workstations.
27
implementation of MPI. On all architectures, the
MPI de(cid:12)ned timer, MPI WTIME, is used to measure rate measure of the communication overhead. Never-
elapsed wall clock time for the main algorithm only. theless, the speedup of the code on all machines is
Thetimetoreadandwriterestart(cid:12)lesandtoperform good. As anticipated, the relative message-passing
pre- and post-processing is not measured although it performance on the dedicated machines (IBM SP2,
mayaccountforasigni(cid:12)cantfractionofthetotaltime. SGI Origin 2000, SGI multiprocessor) is better than
Compiler options include ‘-O3 -qarch=pwr2’ on the on the network of SGI workstations. Also, the per-
IBM SP2and ‘-O2 -n32-mips4’ onthe SGI machines. formance with data exchanged every 20 iterations is
No e(cid:11)ort is made to optimize the single-node perfor- noticeablybetter on the networkof workstationsthan
mance of LAURA on these cache-basedarchitectures. with data exchanged every iteration. However, there
Figures 3 - 6 display the elapsed wall clock times is little in(cid:13)uence of nexch on elapsed time on the
on the various machines. A time based on the dedicated machines which indicates that the commu-
single-node time and assuming a linear speedup equal nication overhead is very low. The degradation in
to the number of nodes is shown for comparison. The performance of the 8 processor runs on the SGI mul-
measured times are less than the comparison time for tiprocessor is due to the load on the machine from
most of the cases as a result of the smaller blocks other users and is not a result of the communication
on each node making better use of the cache. This overhead. Of course, the times (and message-passing
increase in cache performance o(cid:11)sets the communica- e(cid:14)ciency) measured will vary depending on machine
tion penalty. Improving the single-node performance and problem size. Also shown in Fig. 3 is the elapsed
of LAURA on these cache-based architectures would timefromamultitaskingrunwiththeoriginalversion
reduce the single-node times and give a more accu- of LAURA on a CRAY C-90 using 9 CPU’s. This
4of9
Table 2 LAURA parameters - viscous.
Iterations Order njcobian
0-100 1 1
101-300 1 2
301-500 2 10
>500 2 20
a)X-33 b)X-34 Table 3 LAURA parameters - inviscid.
Iterations Order njcobian
0-100 1 1
101-300 1 2
301-900 1 10
901-1100 2 10
>1100 2 20
uration). A baseline solution is generated with nexch
equal to 1. Updating the boundary data every itera-
c)X-33 forebody d)Stardust capsule
tion should mimic the communication between blocks
Fig. 7 Vehicle geometries. in the shared-memory version of LAURA. A second
computation is made with nexch equal to njcobian
shows that performance comparable to current vector since acceptable values for both parameters depend
supercomputers may be obtained on dedicated paral- on transients in the (cid:13)ow. Solutions that are chang-
lel machines (albeit with more processors) using this ing rapidly should update the Jacobian and exchange
distributed-memory version of LAURA. boundary data frequently, while partially converged
solutions may be able to freeze the Jacobian and lag
Convergence
theboundarydataforanumberofiterations. Asimple
The e(cid:11)ect of problem size, gas chemistry, and strategyistolinkthetwoparameters. Convergenceis
boundary data exchange frequency on convergence is measured by the L2 norm de(cid:12)ned by
10,11
examined for four realistic geometries: the X-33
12 N
and X-34 RLV concepts, the X-33 forebody, and 1 (rL(cid:1)rL)
the Stardust sample return capsule forebody.8 All L2 = 2 X 2 (2)
CN L=1 (cid:26)L
four geometries are shown in Fig. 7. A viscous (thin-
layer Navier-Stokes),perfect gas solution is computed
whereCN istheCourantnumber,N isthetotalnum-
over the X-33 and X-33 forebody con(cid:12)gurations. The
ber of cells, rL is the residual vector, and (cid:26)L is the
convergence of an inviscid, perfect gas solution is ex-
local density. All solutions are generated on the IBM
amined using the X-34 vehicle. Nonequilibrium air
SP2.
chemistrye(cid:11)ects on the convergenceand performance
of the distributed-memory version of LAURA are de- X-33
terminedfromaviscous,11-speciesaircalculationover The viscous, perfect gas (cid:13)ow (cid:12)eld is computed over
theStardustcapsule. Forallgeometries,the vehicleis the X-33 RLV con(cid:12)guration (without the wake) to
de(cid:12)ned by the k =1 surface, and the outer boundary demonstrate the ability of the new message-passing
of the volume grid is de(cid:12)ned by k = kmax. version of LAURA to handle large-scale problems in
Each viscous solution is computed with the same a reasonable amount of time. The freestream Mach
sequence of parameters for consistency and is started number is 9.2, the angle of attack is 18.1deg, and the
with all (cid:13)ow-(cid:12)eld variables initially set to their altitude is 48.3 km. The grid size is 192 x 168 x 64
freestream values. Slightly di(cid:11)erent values are used and is divided into 64 blocks of 48 x 42 x 16.
for the inviscid solutions due to low densities on the Figure 8 shows the L2 convergence as a function of
leeside of the vehicle causing some instability when number of iterations. The elapsed wall clock time on
switchingfrom(cid:12)rsttosecondorderaccuracy. Methods the SP2 is 12.7hr. Only the baseline case(nexch = 1)
to speed convergence such as computing on a coarse was computed due to resource limitations. The e(cid:11)ect
grid before proceeding to the (cid:12)ne grid and converging of nexch on convergence will be examined in greater
blocks sequentially beginning at the nose (i.e. block detail on the nose region of this vehicle. The stall
5
marching) arenotused. TherelevantLAURAparam- in convergence after 10000 iterations is due to a limit
eters areshownbelow. Two valuesof nexch areused cycle in the convergence at the trailing edge of the
(exceptfortheruninvolvingthecompleteX-33con(cid:12)g- tip of the canted (cid:12)n. Iterations are continued past
5of9
X-33
106 M¥ =9.2 104 nexch=1
a =18.1deg splitinI-J-K } nexch=njcobian
splitinI-J
ViscousPG
104 192x168x64 102
64nodes-IBMSP2
X-33forebody
102 nexch=1 100 M¥ =9.2
a =18.1deg
L2 12.7hr L2 ViscousPG
100 10-2 64x56x64
16nodes-IBMSP2
10-2 10-4
10-4 10-6
0 4000 8000 12000 16000 0 4000 8000 12000
iterations iterations
a)Convergence as function of number of iterations
Fig. 8 Convergencehistory of viscous, perfect gas
(cid:13)ow (cid:12)eld over X-33 vehicle.
this point to convergethe boundary layerand surface 104 nspelxitcihn=I-1J-K }
heating. splitinI-J nexch=njcobian
asynchronous
X-33 forebody 102
The e(cid:11)ects of boundary data exchange frequency
and block splitting on convergence are evaluated for 100
the nose section of the X-33. This is the same con-
L2
(cid:12)guration used to obtain the timing estimates, and
10-2
freestream conditions correspond to the complete X-
33 vehicle case. The 64 x 56 x 64 grid is (cid:12)rst divided
inthei-,j-,andk-directionsinto16blockscomprised 10-4
of 32 x 28 x 16 cells each. Two cases, nexch = 1 and
nexch = njcobian, are run using this blocking. An-
10-6
other case is computed with the grid divided in the i- 0 1 2 3 4
andj-directionsonlyresultinginblocksof16x14x64 wallclocktime(hr)
cells. Next,theasynchronousrelaxationcapabilitiesof
LAURAaretestedbyreblockingapartiallyconverged b)Convergence as function of time
restart (cid:12)le in the k-direction to cluster work (and it-
Fig. 9 Convergence histories of viscous, perfect
erations) in the boundary layer. Each block has i x j
gas (cid:13)ow (cid:12)eld over X-33 forebody.
dimensionsof32x28,butthek dimensionissplitinto
8, 8, 16, and 32 cells. Blocks near the wall contain 32
the k-direction.
x 28 x 8 cells, while blocks near the outer boundary
have 32 x 28 x 32 cells. Thus, the smaller blocks ac- Figure 9(b) shows the convergence as a function of
cumulate more iterations than the larger outer blocks wall clock time. Because of the low communication
in a given amount of time and should convergefaster. overhead on the IBM SP2, the time saved by mak-
Figure9showsthe convergencehistoryforthis(cid:13)ow ing fewer boundary data exchanges is small. As seen
(cid:12)eld. For viscous solutions, convergence is typically from the timing data, this would not necessarily be
dividedintotwostages. First,the inviscidshocklayer true on a network of workstations where the decrease
develops and then the majority of the iterations are in communication overhead might o(cid:11)set any increase
spentconvergingtheboundarylayer(andsurfaceheat- in number of iterations. Also shown are LAURA’s
ing). Laggingtheboundarydataappearstohavemore asynchronous relaxation capabilities. After 1 hr (and
of an impact on the early convergence of the inviscid 3500 iterations), the outer inviscid layer is partially
featuresofthe(cid:13)owandlessofanimpactonthebound- converged. Restructuring the block structure at this
arylayerconvergence. Thise(cid:11)ectismuchlargerwhen point by splitting the k dimension into 8, 8, 16, and
the blocksaresplit in the k-directionacrossthe shock 32cellsallowstheboundarylayertoaccumulatemore
layer. The communication delay a(cid:11)ects the develop- iterationsandacceleratesconvergence. Theresultisa
ing shock wave as it crosses the block boundaries in 15percentdecreaseinwallclocktimecomparedtothe
6of9
baseline (nexch = 1) case. A similar strategy would
also have accelerated the convergence of the baseline 104 nspelxitcihn=I-1J-K }
nexch=njcobian
case. splitinI-J
103
X-34
102
Thee(cid:11)ectofboundarydataexchangefrequencyand X-34
block splitting on convergence of inviscid, perfect gas 101 M =6.32
¥
a =23deg
(cid:13)owsisinvestigatedfortheX-34con(cid:12)guration(minus
the body (cid:13)ap and vertical tail). Inviscid solutions are L2100 InviscidPG
usefulinpredictingaerodynamiccharacteristicsforve- 10-1 120x152x32
hicledesignandmaybecoupledwithaboundary-layer 32nodes-IBMSP2
techniquetopredictsurfaceheattransferaswell. The 10-2
freestreamMachnumberis6.32,theangleofattackis
10-3
23 deg, and the altitude is 36 km. The grid is 120 x
152x32andis(cid:12)rstdividedinto32blocksof30x38x 10-4
16cells. Thegridisalsosplitinthei-andj-directions 0 1000 2000 3000 4000
iterations
into blocks of 30 x 19 x 32 cells to check the e(cid:11)ect of
block structure on convergence. The convergence his-
tories areshownin Figure10. The aerodynamics(not a)Convergence as function of number of iterations
shown)ofthevehicleareconvergedat4000iterations.
Thespikeinconvergenceat900iterationsiscausedby
104 nexch=1
the switch from (cid:12)rst to second order accuracy. With splitinI-J-K }
nexch=njcobian
splitinI-J
the grid split in all directions, the bas(cid:0)e3line solution 103
(nexch = 1) reaches an L2 norm of 10 at 3300 it-
erationswhilethesolutionwithboundarydatalagged 102
takes 3640iterations. The solution with the grid split
101
in the i- and j-directions requires 3530 iterations. As
showninFig.10(b),thereisacorrespondingdi(cid:11)erence L2100
in run times to reach that convergence level because
the savings from fewer boundary data exchanges are 10-1
small on the SP2. Nevertheless, the e(cid:11)ect of lagging
10-2
the boundary data on convergenceis minimal.
Stardust 10-3
Theconvergenceofanonequilibriumair(11species, 10-4
0 1 2
two temperature), viscous computation is examined
wallclocktime(hr)
for the forebody of the Stardust capsule. The
freestream Mach number is 17 and the angle of at-
b)Convergence as function of time
tackis 10 deg. The gridis 56 x 32 x 60and is divided
into 32 blocks of 7 x 8 x 60 cells each. There are Fig. 10 Convergence histories of inviscid, perfect
no splits in the k-direction. Figure 11 shows the con- gas (cid:13)ow (cid:12)eld over X-34 vehicle.
vergence as a function of iterations and elapsed wall
clock time. Because of the larger number of (cid:13)ow-(cid:12)eld
take advantage of distributed-memory parallel ma-
variables, considerably more data must be exchanged
chines. A standard domain decomposition strategy
between blocks for nonequilibrium (cid:13)ows. Even on a
yields good speedup on dedicated parallel systems,
dedicated parallel machine such as the IBM SP2, the
but the single-nodeperformanceof LAURA on cache-
communication penalty for this particular case has a
based architecturesrequires further study. The point-
signi(cid:12)cant impact on the elapsed time. The baseline
(cid:0)4 implicit relaxation strategy in LAURA is well-suited
case reaches an L2 norm of 10 at 6900 iterations
for parallel computing and allows the communication
compared to 7500 iterations for the nexch = njcobian
overhead to be minimized (if necessary) by reducing
solution. However,thesavingsincommunicationtime
the frequency of boundary data exchanges. The com-
allowsthe nexch = njcobian solution to converge1 hr
munication overhead is greatest on the network of
faster than the baseline case.
workstationsandfornonequilibrium(cid:13)owsduetomore
data passing between nodes. Lagging the boundary
Conclusions databetweenblocksappearstoa(cid:11)ectthedevelopment
The shared-memory, multitasking version of the of the inviscid shock layer more than the convergence
CFD code LAURA has been successfully modi(cid:12)ed to of the boundary layer. Its largest e(cid:11)ect occurs when
7of9
Acknowledgements
104 nexch=1 } splitinI-J
nexch=njcobian The authors wish to acknowledge Peter Gno(cid:11)o of
the Aerothermodynamics Branch at NASA LaRC for
102 Stardust his assistancewith the inner workingsof LAURA and
M =17 Jerry Mall of Computer Sciences Corporation for his
¥
a =10deg help in pro(cid:12)ling LAURA on the IBM SP2.
100 Viscous11-speciesair
L2 56x32x60 References
32nodes-IBMSP2 1Gno(cid:11)o, P. A., \An Upwind-Biased, Point-Implicit Relax-
10-2
ationAlgorithmforViscous,CompressiblePerfect-GasFlows,"
NASATP{2953, Feb.1990.
2
Gno(cid:11)o, P. A., \Upwind-Biased, Point-Implicit Relaxation
10-4 Strategies for Viscous, Hypersonic Flows," AIAA Paper 89{
1972,Jun.1989.
3
Gno(cid:11)o, P. A., \Code Calibration Program in Support of
10-6 the Aeroassist Flight Experiment," Journal of Spacecraft and
0 2000 4000 6000 8000 Rockets,Vol.27,No.2,1990,pp.131{142.
iterations 4
Weilmuenster, K. J. and Greene, F. A., \HL-20 Compu-
tational Fluid Dynamics Analysis," Journal of Spacecraft and
a)Convergence as function of number of iterations Rock5ets,Vol.30,No.5,1993,pp.558{566.
Gno(cid:11)o,P.A.,Weilmuenster,K.J.,andAlter,S.J.,\Multi-
blockAnalysisforShuttleOrbiterRe-EntryHeatingFromMach
24 to Mach 12," Journal of Spacecraft and Rockets, Vol. 31,
104 nneexxcchh==1njcobian} splitinI-J No.3,1994, pp.367{377.
6
Mitcheltree, R. A. and Gno(cid:11)o, P. A., \Wake Flow About
theMarsPath(cid:12)nder EntryVehicle,"Journal of Spacecraft and
102 Rockets,Vol.32,No.5,1994,pp.771{776.
7
Weilmuenster, K. J., Gno(cid:11)o, P. A., Greene, F. A., Riley,
C. J., Hamilton, H. H., and Alter, S. J., \Hypersonic Aero-
100 dynamic Characteristics of a Proposed Single-Stage-to-Orbit
Vehicle," Journal of Spacecraft and Rockets, Vol. 33, No. 4,
L2 1995,pp.463{469.
8
Mitcheltree, R. A., Wilmoth, R. G., Cheatwood, F. M.,
10-2
Brauckmann,G.J.,andGreene,F.A.,\AerodynamicsofStar-
dustSampleReturnCapsule,"AIAAPaper97{2304,Jun.1997.
9
Mitcheltree,R.A.,Moss,J.N.,Cheatwood,F.M.,Greene,
10-4 F.A.,andBraun,R.D.,\AerodynamicsoftheMarsMicroprobe
EntryVehicles,"AIAAPaper97{3658,Aug.1997.
10
Cook, S. A., \X-33 Reusable Launch Vehicle Structural
10-6 Technologies,"AIAAPaper96{4573,Nov.1996.
0 2 4 6 8 10 11
Gno(cid:11)o,P.A.,Weilmuenster,K.J.,Hamilton,H.H.,Olyn-
wallclocktime(hr)
ick,D.R.,andVenkatapathy,E.,\ComputationalAerothermo-
dynamic Design Issues for Hypersonic Vehicles," AIAA Paper
b)Convergence as function of time 97{2473, Jun.1997.
12
Levine,J.,\NASAX-34Program,"MeetingPapersonDisc
Fig.11 Convergencehistoriesofviscous,nonequi- A9710806,AIAA,Nov.1996.
13
librium (cid:13)ow (cid:12)eld over Stardust capsule. Cheatwood, F. M. and Gno(cid:11)o, P. A., \User’s Manual for
theLangleyAerothermodynamicUpwindRelaxationAlgorithm
(LAURA),"NASATM{4674,Apr.1996.
14
Gno(cid:11)o, P. A., \Asynchronous, Macrotasked Relaxation
StrategiesfortheSolutionofViscous,HypersonicFlows,"AIAA
Paper91{1579,Jun.1991.
the grid is split in the direction normal to the vehicle 15
Jayasimha, D. N., Hayder, M. E., and Pillay, S. K., \An
surface. However, restructuring the blocks to cluster
EvaluationofArchitecturalPlatformsforParallelNavier-Stokes
work and iterations in the boundary layer improves Computations,"NASACR{198308,Mar.1996.
16
overall convergence once the inviscid features of the Venkatakrishnan,V.,\ParallelImplicitUnstructuredGrid
(cid:13)ow have developed. These results demonstrate the EulerSolvers,"AIAAPaper94{0759,Jan.1994.
17
ability of the new message-passingversion of LAURA Wong, C.C.,Blottner, F.G.,Payne,J.L.,andSoetrisno,
M., \A Domain Decomposition Study of Massively Parallel
to e(cid:11)ectively use distributed-memory parallelsystems
Computing in CompressibleGas Dynamics," AIAA Paper 95{
for realistic con(cid:12)gurations. As a result, the e(cid:11)ective- 0572,Jan.1995.
18
nessofLAURAasanaerospacedesigntoolisenhanced Borrelli, S., Schettino, A., and Schiano, P., \Hyper-
byitsnewparallelcomputingcapabilities. Infact,this sonicNonequilibriumParallelMultiblockNavier-StokesSolver,"
JournalofSpacecraftandRockets,Vol.33,No.5,1996,pp.748{
new version of LAURA is currently being applied to
750.
theevaluationofvehiclesusedinplanetaryexploration 19
Domel,N.D.,\Research inParallel Algorithmsand Soft-
missions and the X-33 program. wareforComputationalAerosciences,"NAS96-004,Apr.1996.
8of9
20
VanderWijngaart,R.F.andYarrow,M.,\RANS-MP:A
PortableParallelNavier-StokesSolver,"NAS97-004,Feb.1997.
21
Forum,M.P. I., \MPI: Amessage-passing interface stan-
dard," Computer Science Dept. Technical Report CS{94{230,
UniversityofTennessee,Knoxville,TN,1994.
22
Geist,A.,Beguelin,A.,Dongarra,J.,Jiang,W.,Manchek,
R., and Sunderam, V., \PVM 3.0 User’s Guide and Reference
Manual,"Tech.rep.,Feb.1993.
23
Balasubramanian, R., \Modi(cid:12)cation of Program LAURA
to Execute in PVM Environment," Spectrex Report 95.10.01,
Oct.1995.
24
Roe, P. L., \Approximate Riemann Solvers, Parameter
Vectors, and Di(cid:11)erence Schemes," Journal of Computational
Physics,Vol.43,No.2,1981,pp.357{372.
25
Harten,A.,\HighResolutionSchemesforHyperbolicCon-
servation Laws," Journal of Computational Physics, Vol. 49,
No.3,1983,pp.357{393.
26
Yee, H. C., \On Symmetric and Upwind TVD Schemes,"
NASATM{86842,Sep.1985.
27
Gropp, W. and Lusk, E., \User’s Guide for mpich, a
PortableImplementationofMPI,"Tech.Rep.ANL/MCS{TM{
ANL{96/6,ArgonneNationalLaboratory,1996.
28
Burns,G.,Daoud,R.,andVaigl,J.,\LAM:AnOpenClus-
ter Environment for MPI," Tech. rep., Ohio Supercomputing
Center,May1994.
9of9
