HPE Apollo 8000 System User Guide

118 Pages·2015·7.84 MB·English

by  Hewlett Packard Enterprise

HPE Apollo 8000 System
User Guide
Abstract
This guide provides user information for the HPE Apollo 8000 System.
Part Number: 747365-003R
November 2015
Edition: 4
© Copyright 2015 Hewlett Packard Enterprise Development LP
The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services
are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting
an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.
Contents
Overview................................................................................................................................................ 6
Product overview ...................................................................................................................................................... 6
Example configurations ............................................................................................................................................ 6
Component identification ..................................................................................................................... 11
f8000 rack components .......................................................................................................................................... 11
iCDU components .................................................................................................................................................. 13
iCDU display components ........................................................................................................................... 14
Installation and configuration ............................................................................................................... 15
Installation included with service ............................................................................................................................ 15
Installation expectations ......................................................................................................................................... 15
Power components .............................................................................................................................. 17
About the power shelf ............................................................................................................................................. 17
Power shelf front components ..................................................................................................................... 18
Power shelf rear components ...................................................................................................................... 19
Installing the power shelf ............................................................................................................................. 21
Configuring the power shelf ......................................................................................................................... 22
Redundant power supplies ..................................................................................................................................... 23
Redundant power supply LED indicator ...................................................................................................... 24
Installing the redundant power supply ......................................................................................................... 24
Monitoring power usage ......................................................................................................................................... 24
Compute cell ........................................................................................................................................ 26
About the compute cell ........................................................................................................................................... 26
Compute cell components ........................................................................................................................... 27
Compute cell numbering.............................................................................................................................. 28
Management module connectors and LED indicators ................................................................................. 29
Fan numbering ............................................................................................................................................ 30
Installing the management module ........................................................................................................................ 35
Utility module ....................................................................................................................................... 37
About the utility module .......................................................................................................................................... 37
Utility module products ................................................................................................................................ 38
HPE Apollo 8000 UM1 ........................................................................................................................................... 39
Utility module front components .................................................................................................................. 39
Utility module rear components ................................................................................................................... 41
HPE Apollo 8000 UM2 ........................................................................................................................................... 42
Utility module front components .................................................................................................................. 43
Utility module rear components ................................................................................................................... 45
Installing the utility module ..................................................................................................................................... 46
Configuring the utility module ................................................................................................................................. 47
Thermal components ........................................................................................................................... 48
About the thermal components .............................................................................................................................. 48
HEX ........................................................................................................................................................................ 49
HEX components and connectors ............................................................................................................... 49
Thermal controller .................................................................................................................................................. 50
Thermal controller components and connectors ......................................................................................... 50
Updating the thermal controller firmware..................................................................................................... 50
Water walls ............................................................................................................................................................. 51
Water wall components and connectors...................................................................................................... 51
Rack door sensors and locks ................................................................................................................................. 52
Activating and deactivating door locks ........................................................................................................ 52
Contents 3
Infiniband switch .................................................................................................................................. 53
About the infiniband switch ..................................................................................................................................... 53
Infiniband switch front components and connectors.................................................................................... 54
Infiniband switch port numbering ................................................................................................................. 54
Infiniband switch LED indicators.................................................................................................................. 55
Infiniband switch rear components and connectors .................................................................................... 56
Installing the infiniband switch ................................................................................................................................ 56
Safety components .............................................................................................................................. 57
About the safety components ................................................................................................................................. 57
Leak detection sensors .......................................................................................................................................... 57
Temperature sensors and automatic door release mechanism ............................................................................. 57
(Optional) ARPO .................................................................................................................................................... 58
Cabling ................................................................................................................................................ 59
Cabling overview .................................................................................................................................................... 59
Cabling when the HPE Apollo 8000 UM1 is installed ............................................................................................. 59
Connecting the utility module to the power shelves .................................................................................... 59
Connecting the utility module to the management modules ........................................................................ 60
Cabling when the HPE Apollo 8000 UM2 is installed ............................................................................................. 61
Connecting power........................................................................................................................................ 61
Connecting communication ......................................................................................................................... 62
Connecting the f8000 rack to the iCDU rack ............................................................................................... 65
Connect the CMU solution hardware ..................................................................................................................... 67
iCDU .................................................................................................................................................... 69
About the iCDU ...................................................................................................................................................... 69
iCDU setup ............................................................................................................................................................. 69
Powering on the iCDU ................................................................................................................................. 69
Powering off the iCDU ................................................................................................................................. 74
BMS network communication ...................................................................................................................... 74
iCDU display ........................................................................................................................................................... 77
Password security level ............................................................................................................................... 77
Screen time-out ........................................................................................................................................... 77
Always available .......................................................................................................................................... 77
Always visible .............................................................................................................................................. 80
iCDU control ........................................................................................................................................................... 81
Control switch .............................................................................................................................................. 81
iCDU isolation valves................................................................................................................................... 82
Vacuum pressure ........................................................................................................................................ 83
Differential pressure .................................................................................................................................... 83
Pump and IT supply out water temperature ................................................................................................ 83
iCDU maintenance ................................................................................................................................................. 84
Water testing and maintenance schedule ................................................................................................... 84
Condensation management ................................................................................................................. 85
Condensation management ................................................................................................................................... 85
Frost damage ......................................................................................................................................................... 85
Software and configuration utilities ...................................................................................................... 86
HPE Insight Cluster Management Utility ................................................................................................................ 86
HPE Advanced Power Manager ............................................................................................................................. 86
Troubleshooting ................................................................................................................................... 87
HPE Apollo 8000 System Troubleshooting ............................................................................................................ 87
Rack and mechanical issues ....................................................................................................................... 87
Power issues ............................................................................................................................................... 88
Temperature and cooling issues ................................................................................................................. 90
Communication issues ................................................................................................................................ 91
iCDU issues ................................................................................................................................................. 93
iCDU alarm troubleshooting ................................................................................................................................... 93
Using HPE APM for sensor monitoring .................................................................................................................. 95
Contents 4
Specifications ...................................................................................................................................... 98
Physical specifications ........................................................................................................................................... 98
Electrical specifications .......................................................................................................................................... 98
Thermal and air flow performance ........................................................................................................................ 100
Environmental specifications ................................................................................................................................ 100
Safety considerations......................................................................................................................... 101
Important safety information ................................................................................................................................. 101
Symbols on equipment ......................................................................................................................................... 101
Warnings and cautions ......................................................................................................................................... 101
Support and other resources.............................................................................................................. 103
Accessing Hewlett Packard Enterprise Support ................................................................................................... 103
Information to collect ................................................................................................................................. 103
Accessing updates ............................................................................................................................................... 103
Websites ............................................................................................................................................................... 103
Customer Self Repair ........................................................................................................................................... 104
Remote support .................................................................................................................................................... 112
Warranty and regulatory information .................................................................................................. 113
Warranty information ............................................................................................................................................ 113
Regulatory information ......................................................................................................................................... 113
Safety and regulatory compliance ............................................................................................................. 113
Belarus Kazakhstan Russia marking ......................................................................................................... 113
Turkey RoHS material content declaration ................................................................................................ 114
Ukraine RoHS material content declaration .............................................................................................. 114
Acronyms and abbreviations .............................................................................................................. 115
Documentation feedback ................................................................................................................... 116
Index .................................................................................................................................................. 117

Contents 5
Overview
Product overview
The HPE Apollo 8000 System is a high-density, energy-efficient, sustainable high-performance
computing solution that uses an innovative warm-liquid cooling technology to fuel the future of
supercomputing. It offers hundreds of teraflops per rack to accelerate results, with a modular, rack-based
system that is easy to install, maintain, and monitor. The hot water "waste heat" can be recycled to heat
the data center efficiently.
The Apollo 8000 System comprises the following components:
• One or more HPE 8000 iCDU Racks and their associated onboard iCDU control system
• One or more f8000 racks and their associated control system
• Primary plumbing interface assembly kit that connects the individual iCDUs to the facility water
system
• Secondary plumbing system that connects the individual iCDUs to the f8000 racks. This secondary
closed loop system provides cooling for the f8000 rack and isolates the f8000 rack from untreated,
potentially incompatible primary water.
• Drain and leak detection system
The iCDU performs the following functions:
• Accepts water supplied from the facility water system
• Returns heated water back to the facility water system
• Supplies cooled water to the f8000 racks through the secondary plumbing system
• Accepts heated water returned from the f8000 racks
• Maintains the preset water temperature and pressure in the secondary plumbing system
• Monitors system health
• Isolates primary and secondary water in case of a leak

Example configurations
The Apollo 8000 System is a self-contained IT product integrated with a closed-loop cooling system that
offers two configuration options.
The following figures demonstrate examples of the f8000 rack (1) to iCDU (2) ratio. The doors are
removed for clarity.

CAUTION: To reduce the risk of condensation and avoid damage to the equipment when
operating with chilled water, keep the rack doors closed during operation.

Example single iCDU configuration (with up to four f8000 racks)
Overview 6
• Front view

Item Description
1 HPE Apollo f8000 Racks
2 HPE Apollo iCDU Racks

Overview 7
• Top view

Item Description
1 Facility water in
2 Facility water out
3 Secondary water out from f8000 rack
4 Secondary water in to f8000 rack
5 Heat exchanger

Example redundant iCDU configuration (with up to four f8000 racks)
Overview 8
• Front view

Item Description
1 Apollo f8000 Racks
2 Apollo iCDU Racks

Overview 9
• Top view

Item Description
1 Facility water in
2 Facility water out
3 Secondary water out of f8000 rack
4 Secondary water in to f8000 rack
5 Heat exchanger

The iCDU transfers heat to the facility water plant and has a secondary loop that provides cooling to the
servers in the IT racks.
Overview 10