DTIC ADA505907: NAVO MSRC Navigator. Spring 2006

3.8 MB·English

by  Defense Technical Information Center

Report Documentation Page Form Approved
OMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington
VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it

does not display a currently valid OMB control number.
1. REPORT DATE 3. DATES COVERED
2006 2. REPORT TYPE 00-00-2006 to 00-00-2006
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
NAVO MSRC Navigator. Spring 2006
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
Naval Oceanographic Office (NAVO),Major Shared Resource Center REPORT NUMBER
(MSRC),1002 Balch Boulevard,Stennis Space Center,MS,39522
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)
11. SPONSOR/MONITOR’S REPORT
NUMBER(S)
12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution unlimited
13. SUPPLEMENTARY NOTES
14. ABSTRACT

15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF
ABSTRACT OF PAGES RESPONSIBLE PERSON
a. REPORT b. ABSTRACT c. THIS PAGE Same as 28
unclassified unclassified unclassified Report (SAR)
Standard Form 298 (Rev. 8-98)
Prescribed by ANSI Std Z39-18
The NAVO MSRC is undergoing a carefully be configured with two gigabytes of memory
planned series of enhancements which, when per processor, IBM's “Federation” inter-
completed by Fall 2006, will provide one of the processor switch fabric, and IBM's Global
most capable, productive, and balanced HPC Parallel File System (GPFS), all of which will
environments ever fielded at this MSRC. These facilitate the execution of tremendously large
enhancements substantially boost the computational applications and also diverse mixes of large
capabilities and resilience of the MSRC across applications — applications which typically run
both the classified and unclassified High as DoD Challenge Projects. To supplement this
Performance Computing (HPC) environments enormous computational capability, we continue
we support for the Department of Defense
(DoD) HPC Modernization Program (HPCMP).
The most significant enhancements will be the
Enhancing the MSRC
addition of two very large IBM HPC systems,
both of which will be based upon IBM's
to Serve You Better
POWER5 processor technology:
(cid:88)KRAKEN (the 3000-processor unclassified
IBM POWER4+), one of the most successful
and requested HPC systems within the
NAVO MSRC, will be joined by an unclassified
to enhance and optimize the internal mass
3000-processor IBM POWER5+ system
storage and networking capabilities of the
(named BABBAGE).
MSRC for both performance and resilience.
(cid:88)ROMULUS (the existing 500-processor IBM
Finally, most of you are aware that the HPCMP
POWER4+ system) will be transitioned to
may be sustaining significant operating budget
the classified environment and will be joined
cuts beginning in FY07. The six HPC centers
by a 1900-processor IBM POWER5+ system
within the program are slated to receive the
(named PASCAL) which will replace the
bulk of these cuts. Please be assured that our
existing classified IBM POWER4 system
primary goal throughout this budget adjustment
(MARCELLUS) when it is retired in early
process will be the maintenance of a premier
Fiscal Year 2007 (FY07).
HPC environment with the support you have
When all of these upgrades are complete, the come to expect from all of the centers. We
effective computing power of the NAVO MSRC invite you, the DoD user community, to let
will be essentially tripled, as measured by us continue to assist you in bringing this
sustainable performance on the HPCMP cutting-edge capability to bear in support of
benchmark suite. All four of these systems will your HPC needs.
2 SPRING 2006 NAVO MSRC NAVIGATOR
Contents
The Naval Oceanographic Office (NAVO)
Major Shared Resource Center (MSRC):
Delivering Science to the Warfighter
The NAVO MSRC provides Department of
Defense (DoD) scientists and engineers with high The Director’s Corner
performance computing (HPC) resources,
including leading edge computational systems, 2 Enhancing the MSRC to Serve You Better
large-scale data storage and archiving, scientific
visualization resources and training, and expertise
Feature Articles
in specific computational technology areas (CTAs).
These CTAs include Computational Fluid
Dynamics (CFD), Climate/Weather/Ocean 4 Five Dogs, a Hurricane, and Two IBM POWER5+s:
Modeling and Simulation (CWO), Environmental A Personal and Professional Katrina Experience
Quality Modeling and Simulation (EQM),
Computational Electromagnetic and Acoustics 5 Drag Reduction by Microbubbles in a Spatially-
(CEA), and Signal/Image Processing (SIP).
Developing Turbulent Boundary Layer: Reynolds
Number Effect (HPCMP/CAP)
NAVO MSRC
Code N7 9 Practical Parallel Graphics and Remote Visualization
1002 Balch Boulevard
12 Mapping the Seabed on an Absolute Reference Frame
Stennis Space Center, MS 39522
System Using the Real-Time GIPSY (RTG) Global
1-800-993-7677 or
[email protected] Differential GPS and RTK Positioning
The Porthole
22 Visitors to the Naval Oceanographic Office
Major Shared Resource Center
24 Boy Scout Jamboree
NAVO MSRC Navigator Navigator Tools and Tips
www.navo.hpc.mil/Navigator
25 Hints for Choosing “Consumable CPUs” and
NAVO MSRC Navigator is a biannual technical “Consumable Memory” Resource Requests on KRAKEN
publication designed to inform users of the news, and ROMULUS
events, people, accomplishments, and activities of
the Center. For a free subscription or to make
Upcoming Events
address changes, contact NAVO MSRC at the
above address.
27 Coming Events
EDITOR:
Gioia Furness Petro, [email protected]
DESIGNERS:
Kerry Townson, [email protected]
Lynn Yott, [email protected]
Any opinions, conclusions, or recommendations in
this publication are those of the author(s) and do
not necessarily reflect those of the Navy or NAVO
MSRC. All brand names and product names are
trademarks or registered trademarks of their
respective holders. These names are for
information purposes only and do not imply
endorsement by the Navy or NAVO MSRC.
Approved for Public Release
Distribution Unlimited
NAVO MSRC NAVIGATOR SPRING 2006 3
Five Dogs, a Hurricane, and Two IBM POWER5+s:
A Personal and Professional Katrina Experience
Dave Cole, Government NAVO MSRC User Support Lead
Articles in the Fall 2005 edition of the Navigator reported country. This brings our pet count to five small, frisky
the unprecedented effects of Hurricane Katrina on the dogs!
Naval Oceanographic Office Major Shared Resource (cid:88)Sun., 28 Aug.: Katrina now a Category (CAT) 5
Center (NAVO MSRC) and employees. The log that headed for New Orleans/West Mississippi (MS) Coast.
follows, which records my evacuation from the Gulf Coast Evacuated to Minden, LA (where my parents live).
and eventual return, provides a more personal snapshot of Boarded the frisky dogs.
events that is in a small way representative of the ordeals
(cid:88)Mon., 29 Aug.: Held second evaluation conference call
that they experienced and overcame.
at 1330 Central Time as scheduled. Welcome surprise
But first, a little background. In January 2005 I became the that my dad, who avoids technology, has a speaker
NAVO MSRC Technical Insertion for Fiscal Year 2006 (TI- phone in the kitchen! Used the kitchen table as my
06) Usability Team Chair. As such, I was charged with command post. Northeast quadrant of Katrina plows
keeping the NAVO MSRC Usability Team process on into the west MS Coast with a storm surge greater than
schedule as part of the Center's acquisition of two large 20 feet (my house is at a 12-foot elevation). Bought a
IBM POWER5+clusters through the Department of Defense Universal Serial Bus (USB) compatible keyboard at
(DoD) High Performance Computing Modernization Program Wal-Mart for my laptop and began development of the
(HPCMP) TI-06 acquisition effort. Usability Brief.
As the NAVO MSRC TI-06 Usability Team Chair, my (cid:88)Tue., 30 Aug.: Moved into Best Western for Digital
participation began with a meeting in January 2005 with Subscriber Line (DSL) access. Completed the draft
the TI-06 Performance Team and essentially culminated Usability Presentation and passed the ball to Tom
with a presentation to the TI-06 Collective Acquisition Team Crimmins of the Army Research Laboratory (ARL) at
(CAT) in November 2005. Keeping the Usability Team 22:24 so I could begin planning the trek home to pick
process on schedule was especially challenging as Hurricane
Katrina struck during the last week of the Phase I analysis.
(cid:88)Fri., 26 Aug.: Held first Usability evaluation conferencecall Continued Page 21
as scheduled. Departed work under normal
hurricane condition status—no preparation needed.
(cid:88)Sat., 27 Aug.: Katrina now a threat—spent the day
cutting plywood shutters for my house and
preparing for evacuation. Liberated two beagles
from local kennel for a friend who was out of the
Five (usually) frisky dogs.
4
Drag Reduction by Microbubbles in a Spatially-
Developing Turbulent Boundary Layer: Reynolds
Number Effect (HPCMP/CAP)
Antonino Ferrante and Said Elghobashi, Department of Mechanical and Aerospace Engineering, University of
California, Irvine
INTRODUCTION framework. The bubble acceleration Time integration in this simulation was
equation, on the other hand, is solved performed using the second-order
Experimental evidence during the past
for each bubble to track its trajectory Adams-Bashforth scheme. The
three decades indicates that the injection
in time.4, 5 discretized Poisson equation for
of gaseous microbubbles (diameter
pressure was solved using a cosine
ranging from 1 to 1000 microns, and The governing equations of the fluid
transform in the streamwise direction,
at a relatively large volume fraction motion account for the instantaneous
a Fast Fourier Transform (FFT) in the
(up to F = 0.7)) into a liquid turbulent
n local volume fraction of the bubbles.
boundary layer over a flat plate1, 2or spanwise direction, and Gauss
The bubble equation of motion
over axisymmetrical bodies3can reduce elimination in the wall-normal direction.
includes terms representing the added
the skin friction by as much as 80 The discrete cosine and Fourier
mass, carrier fluid inertia, Stokes drag,
percent from its value without bubble transforms were computed using the
buoyancy, and lift force. The governing
injection. However, the basic physical Fastest Fourier Transform in the West
mechanisms responsible for that equations4, 5were discretized in space (FFTW) C subroutine library.6
reductionwere not yet fully understood. using a second-order finite difference
scheme—except for the mean advection
This article discusses the physical
mechanisms responsible for the terms, which were evaluated via a
reduction of skin friction in a fifth-order upwind differencing scheme. Continued Next Page...
microbubble-laden, Spatially-Developing
Turbulent Boundary Layer (SDTBL)
over a flat plate4, and the effects of
increasing Reynolds number on drag Fluid + Bubbles
reduction.5This discussion is based
on the results of the author's Direct
Numerical Simulations (DNS) of a
microbubble-laden SDTBL. These
simulations were performed on High Z
Performance Computing (HPC) highly
scalable supercomputers (CRAY T3E
<U > (x, z)
and IBM Power4+ (KRAKEN)) at the
1
Naval Oceanographic Office
Y
(NAVOCEANO) Major Shared
Resource Center (MSRC).
g
MATHEMATICAL DESCRIPTION
Figure 1 shows a schematic of the
X
SDTBL flow where the gravitational
acceleration vector is perpendicular to
Wall
the wall and pointing downward. The
DNS used in this simulation employs
the Eulerian-Lagrangian approach to
solve the fluid continuity and
momentum equations in an Eulerian Figure 1.Schematic of microbubble-laden turbulent boundary layer flow
over a flat wall.
NAVO MSRC NAVIGATOR SPRING 2006 5
CAP PARALLEL DNSBLB is more than ten times faster scalability of the code. Figure 2 shows
PERFORMANCE TESTS than its T3E version. The scalability the CPU time in seconds per processor,
runs of the DNS code were performed per time step needed for the integration
During the past four years the authors
using up to 1024 processors for two of the governing equations of both the
have used their newly-developed
different computational meshes: a fluid and bubbles, versus the number
parallel code (DNSBLB, written in
coarse mesh of 256x256x96 = 6x106 of processors used on KRAKEN.
FORTRAN 90/MPI), which performs a
grid points, and a fine mesh of Different lines correspond to different
DNS of a microbubble-laden SDTBL.
4, 5, 7, 8, 9DNSBLB is parallelized with 1024x1024x96 = 101x106grid computationalmeshes and numbers
a One-Dimensional (1D) domain points; and for four different numbers of bubbles. The CPU time monotonically
decreases as the number of processors
decomposition in the spanwise of bubbles 100 thousand, 16 million,
increase (see Figure 2) for all the tests
y-direction (j) of the three-dimensional 100 million, and 200 million.
performed.
computational domain.*DNSBLB was The scalability tests were performed in
Figure 2 also shows that for the coarse
originally written for the CRAY T3E. a one-way coupling regime, i.e., no
mesh with 100 thousand bubbles, the
During the Capability Applications effects of bubble on turbulence were
slope of the line (CPU time versus
Project (CAP) 2004, the CRAY T3E accounted for. The two-way coupling
number of processors) is close to
version of DNSBLB was converted to increases the Central Processing Unit
-0.1, whereas for the fine mesh with
run on KRAKEN. The IBM version of (CPU) time but does not worsen the
200 million bubbles, the slope is close
to -1.
The slopes for the other tests are
Coarse mesh: 256x256x96=6.2x106 between -0.1 and -1. This means that
Fine mesh: 1024x1024x96=101x106 the performance of the DNS code
improves as the number of grid points
of the computational mesh and the
p
e number of bubbles simulated increase.
t
e S 101 slope = 1 Furthermore, for the fine mesh the
m code shows a nearly ideal scalability
Ti since the line in Figure 2 has a slope
r
e close to -1.
P
&
r DRAG REDUCTION BY
o
s MICROBUBBLES: REYNOLDS
s
ce NUMBER EFFECT
o
Pr The DNS results4for the microbubble-
er 100 laden SDTBL for RrV= 1430 with
P
) volume fraction ranging from Fn=
c
e 0.001 to 0.02 show that the presence
s
( slope = -0.1 of bubbles results in a local positive
me divergence of the fluid velocity, DD.U >
U ti 0. This creates a positive mean
P 10-1 velocity, <U3>, normal to (and away
C
from) the wall which, in turn, reduces
100 101 102 103
the mean streamwise velocity and
Number of Processors displacesthe quasi-streamwise
longitudinal vortical structures away
from the wall as in Figure 3. This
Coarse mesh & 100K Bubbles Fine mesh & 200M Bubbles
Coarse mesh & 16M Bubbles slope = -1 displacement has two main effects:
Coarse mesh & 100M Bubbles slope = -0.1
Fine mesh & 100M Bubbles (cid:88)It increases the spanwise gaps
between the wall streaks
associatedwith the sweep events
Figure 2.Scalability on IBM P4+ (KRAKEN) of DNS code. and reduces the streamwise
*If np is the number of processors and N, even multiple of np, is the number of grid points in the jdirection, then each processor makes calculation
on N=npplanes (x-zplanes) of the computational domain.
6 SPRING 2006 NAVO MSRC NAVIGATOR
velocity in these streaks, thus per processor and 124 gigabytes of the same amount of reduction in skin
reducing the skin friction. maximum total memory. friction at higher Reynolds number.
(cid:88)It moves the location of peak The DNS results5show that increasing
Reynolds stress production away Reynolds numbers from 1430 to 2900 SUMMARY
from the wall to a zone of a smaller decreases the percentage of drag This article has briefly discussed the
transverse gradient of the mean reduction from 22 to 19 percent.
physical mechanisms of drag reduction
streamwise velocity (i.e., smaller Increasing RrV“squeezes” the quasi-
by microbubbles and the Reynolds
mean shear), thus reducingthe streamwise vortical structures toward
number effect.4, 5
production rate of turbulence the wall, whereas microbubbles “push
Furthermore, it reports some details
kinetic energy and enstrophy. them away” from the wall.
on the simulations of parallel DNS
During CAP 2004 the authors were The net result of these two opposing
code and its scalability on the
able to simulate the microbubble- effects determines the amount of skin
NAVOCEANO MSRC KRAKEN system.
laden SDTBL for RrV= 2900 with friction reduction by the microbubbles.
volume fraction F = 0.01 and bubble The displacement action by the In conclusion, the CAP program has
n
diameter of 40um using a computational microbubbles is a result of the local considerably helped the author's drag
mesh of 1024x512x128 = 67x106 positive velocity divergence, DD .U, reduction by microbubbles research
grid points and 29 million bubbles. created by their concentration gradients. to explain the effect of Reynolds
Thus, the volume fraction of bubbles number on drag reduction by allowing
The computations were performed
on 512 processors of the KRAKEN that is responsible for the reduction of the use of a large number of processors
system. The bubble-laden flow skin friction in a low Reynolds number and extended CPU hours which were
simulations required 9.6 CPU hours SDTBL is not sufficient to produce not allowed in the “standard” queues.
Single-Phase Flow Bubble-Laden Flow
Quasi-Streamwise Vortex Quasi-Streamwise Vortex Displaced
Away from Wall
‘sweep’ ‘ejection’ ‘sweep’ ‘ejection’
Induced Vertical
Fluid Velocity
g g by Velocity
DivergenceEffect
Wall Wall
High Speed Low Speed Reduced Increased
Streak Streak High Speed Low Speed
Streak Streak
Reduced
Skin Friction
Figure 3.Schematic of the drag reduction mechanism in a microbubble-laden SDTBL.
NAVO MSRC NAVIGATOR SPRING 2006 7
Acknowledgements
This work was supported by Office of Naval Research (ONR) Grant No. N00014-05-1-0059 under the
direction of Dr. Patrick Purtell. The computations were performed on the IBM Power4+ (KRAKEN)
located at the NAVO MSRC at the John C. Stennis Space Center (Mississippi), and on CRAY T3E (MD)
located at the U.S. Army High Performing Computing Research Center (AHPCRC, Minnesota). The
authors thank the outreach group at the NAVOCEANO MSRC and the support group at AHPCRC for
allowing jobs with CPU time limits larger than those allowed in the “standard” queues.
References
1. Madavan, N.K., S. Deutsch, and C.L. Merkle, “Reduction of Turbulent Skin Friction By
Microbubbles,” Physics of Fluids, 27: 356-363, 1984.
2. Pal, S., C. L. Merkle, and S. Deutsch, “Bubble Characteristics and Trajectories in a Microbubble
Boundary Layer,” Physics of Fluids, 31(4): 744-751, 1988.
3. Deutsch, S., and S. Pal, “Local Shear Stress Measurements on an Axisymmetrical Body in a
Microbubble-Modified Flow Field,” Physics of Fluids, 2: 2140-2146, 1990.
4. Ferrante, A., and S. Elghobashi, “On the Physical Mechanisms of Drag Reduction in a Spatially
Developing Turbulent Boundary Layer Laden with Microbubbles,” Journal of Fluid Mechanics, 503:
345-355, 2004.
5. Ferrante, A., and S. Elghobashi, “Reynolds Number Effect on Drag Reduction in a Microbubble-
Laden Spatially Developing Turbulent Boundary Layer,” Journal of Fluid Mechanics, 543: 93-106,
2005.
6. Frigo, M., and S.G. Johnson, “The Design and Implementation of FFTW3,” Proceedings of the IEEE,
93(2): 216-231, 2005.
7. Ferrante, A., and S. Elghobashi, “A Robust Method for Generating Inflow Conditions for Direct
Simulations of Spatially-Developing Turbulent Boundary Layers,” Journal of Computational Physics,
198:372-387, 2004.
8. Ferrante, A., S. Elghobashi, P. Adams, M. Valenciano, and D. Longmire, “Evolution of Quasi-
Streamwise Vortex Tubes and Wall Streaks in a Bubble-Laden Turbulent Boundary Layer Over A Flat
Plate,” Physics of Fluids, 16(9): S2, 2004.
9. A. Ferrante, “Reduction of Skin-Friction in a Microbubble-Laden Spatially Developing Turbulent
Boundary Layer Over a Flat Plate,” PhD thesis, University of California, Irvine, 2004.
8 SPRING 2006 NAVO MSRC NAVIGATOR
Practical Parallel Graphics and Remote Visualization
Sean Ziegeler, Visualization Software Engineer, NAVO MSRC VADIC
The purpose of scientific visualization disks before visualization or transferring improved performance. This
is, quite often, to analyze the output of it “on-the-fly” during visualization improvement is because a parallel
a computational model. As computational using a remote file system. file system consists of multiple disks
power increases, model output grows The next technique, parallel graphics, spread out over many or all nodes of
in size, which introduces new obstacles uses large graphics systems (such as the cluster. An example at the Naval
that prevent users from analyzing their clusters) to render in parallel fashion Oceanographic Office (NAVOCEANO)
data—especially if they are not collocated by distributing the load across multiple Major Shared Resource Center (MSRC)
with the computing resources. Graphics Processing Units (GPU's). is the /scrfile system on the visualization
The traditional approach to scientific Finally, remote visualization can deliver cluster, SEAHORSE.
visualization from remote sites has the rendered images, reduced now If the data will not fit on /scr, will not
been to transfer the output data from merely to pixels (and much smaller be visualized repeatedly, or won't
the computational system to the than the original data), from the require the high-speed access of a
user's local site as shown in Figure 1. graphics system to the user's local parallel file system, then the Secure
This is less feasible with larger data system. This new approach is shown Shell File System (SSHFS) is an
because it may be too large to (1) in Figure 2. excellent tool for accessing data on a
transfer over the network between separate system as if it were local.
sites in a reasonable amount of time, REMOTE DATA ACCESS SSHFS uses the Secure Shell (SSH)
(2) fit on the user's system disk program that is already installed on all
Staging the data on a visualization
space, or (3) render on the user's MSRC systems to facilitate authenticated
system is very similar to transferring
local graphics computer. and encrypted file access.
the data from the computational
To overcome all three problems, a system to the user's local system. The difference between SSHFS and a
combination of techniques must be This is the preferred method if the traditional file transfer is that the remote
utilized: computational data access, data set is to be visualized repeatedly, files will appear to be like any other
parallel graphics, and remote visualization. is not too large to fit on the file on the local system. Command
Example 1 shows how to “mount”
Computational data access involves visualization system, and requires
or begin, briefly utilize, and end an
the transfer of the data between a high performance access.
SSHFS session.
computational and a visualization The result of staging the data in
system, either by staging it on large parallel on a graphics cluster is Continued Next Page...
Figure 1.Traditional scientific visualization by transferring data to user's site.
NAVO MSRC NAVIGATOR SPRING 2006 9