flight-appliance-docs Documentation

148 Pages·2016·5.3 MB·English

flight-appliance-docs Documentation
Release 1.0
Alces Flight Ltd
Jul 25, 2019

Using your Flight Compute cluster
1 License 3
2 Prerequisites 5
i
ii
flight-appliance-docsDocumentation,Release1.0
This site holds documentation designed to help users create a simple research compute environment using popular
public and private cloud platforms. As well as launching and accessing the environment, guides and tutorials are
includedtohelpend-usersinstallsoftwareapplications,managetheirdataandrundifferentworkloads.
UsingyourFlightComputecluster 1
flight-appliance-docsDocumentation,Release1.0
2 UsingyourFlightComputecluster
1
CHAPTER
License
ThisdocumentationisreleasedundertheCreative-Commons: Attribution-ShareAlike4.0Internationallicense. The
AlcesFlightComputesoftwareisreleasedunderthetermsoftheAlcesFlightEULA.Pleasereadtherelevantlicense
textbeforeproceedingtouse.
3
flight-appliance-docsDocumentation,Release1.0
4 Chapter1. License
2
CHAPTER
Prerequisites
We recommend that users wishing to use Flight Appliances have basic Linux skills. The ability to move about in a
filesystem,copyanddeletefiles,readandeditfilesonthecommand-linewillbeneededinordertogetthebestoutof
theFlightsoftware.
2.1 Getting started as quickly as possible
Thisdocumentationisdesignedtoprovideaquick-startguidetoyourpersonalFlightComputecluster,butifyou’ve
used clusters before then you might just need a super-quick guide to help you get started. Here are just the basics,
alongwiththingsyoushouldprobablylookatinmoredetailwhenyouhavetime.Usefulcommandsareincludedwith
eachstep-pleaserefertothefulldocumentationforfurtherdetails.
1. LaunchyourpersonalAlcesFlightComputeclusteronAWSoracompatibleOpenStackprivatecloud. Touse
theAWSplatform,usetheprovidedCloudFormation(CFN)templatetostartapersonalclusterinanewVPC.
Youwillbeaskedvariousquestionsabouthowyouwantyourclustertolook-mostoptionsareexplained,with
sensibledefaultsshown.
2. LogintoyourclusterviaSSHusingyourchosenusername,andtheSSHkeyregisteredwithyourcloudprovider.
YourcloudserviceshouldreporttheaccessIPaddresstologintoonceyourclusterislaunched. Ifyouenabled
clusterauto-scaling,computenodesareautomaticallyadded/removedbasedonthestatusofyourjob-scheduler
queue. Use alces configure autoscaling disable to turn off scaling if you’ll be running jobs
manually.
3. Start a graphical desktop session with the command alces session start gnome. Connect using a
VNCclient,withtheone-timepasswordshownwhenyoustartedthesession. Multipleuserscanconnecttothe
samesessionforcollaborativeprojects. Usethexrandrcommandtochangethescreenresolutioninarunning
desktopsession.
4. Your user has full sudo access, and yum is configured to install Linux packages. Load the PDSH module
with the command module load services/pdsh, then use the nodeattr -n nodes command to
see your list of compute node hostnames, and pdsh -g nodes <command> to run commands across all
nodesinyourcluster. TheFlightComputeSoloproductisdesignedtosupportasingleuser,althoughmultiple
userscanloginusingthesameaccount.
5
flight-appliance-docsDocumentation,Release1.0
5. Your cluster has an NFS shared filesystem mounted across all nodes, which your home-directory is part of.
CopydatatoandfromtheclusterusingSCP/SFTP.Thealces storagecommandsprovideaccesstoobject
storage services including S3, Swift and Dropbox. Clusters can also be configured to use other storage types
includingparallelfilesystemsandsharedfilers,dependingonyourcloudplatform.
6. Usethealces gridwarecommandtoviewandinstallsoftware.Themainrepositorylistsstablepackages,
withalargerlistofapplicationsavailableaspartofthevolatilerepository.
7. Your cluster comes with an installation of a batch job scheduler; the default is SLURM, and a range of other
schedulersarealsoavailable.Usethealces templatecommandforalistoftemplatejob-scriptstogetyou
started.
8. Terminatetheclusterstackviayourcloudproviderwhenyou’vefinishedworking. Remembertocopyanydata
youwanttokeepofftheclustertosecurestoragebeforeterminatingit.
2.2 What’s New
2.2.1 2017.2
• Choosethejobschedulertobeconfiguredonyourcluster(SLURM,GridEngine,OpenLava,TORQUEorPBS)
• PreselectGridwaredepotstoinstallonthecluster-yoursoftwareisreadytousefromthewordgo
• Additionaloptionsforformattingandstructuringthestoragelayoutofthecluster,including:
– EBSstoragevolumetype
– Separatingpartitionsforhomeandapplicationsdirectories
– Setingencryptionpoliciesonvolumes
– Configuringlocalscratchspace
• ExtendedAlcesGridwareDockerfeatures-including
– Imagesharing
– Localregistries
– MPIcontainerexecution
– Customvolumemounts
• Thenewc5,h1,p3,m5&x1eAWSinstancesarenowavailableforselection
• FlightcanbelaunchedinthenewAWSregioneu-west-3(Paris)
• PersonalGridware-installanyoftheGridwareappstoyouruserdirectory
• Newclusterdashboard-FlightLaunch-replacesthepreviousdashboard
2.2.2 2017.1
• Newdefaultjobscheduler-SLURM
• Theloginnodeisnowsettoamorepowerfulnodetype
• Evenmorenodetypestochoosefrom
• TheAlcesGridwarecommandnowsupportsbuildingofDockercontainers
• Extendedcustomizationprofilefunctionality-setupyourenvironmenthowyouwant
6 Chapter2. Prerequisites