Workload Management and Application Placement for the Cray Linux Environment™

Changes to this Document

Revised information:

Table of Contents

1. System Overviews
1.1. Cray System Features
2. Running Applications
2.1. Using the aprun Command
2.1.1. ALPS Application Environment Variables
2.2. Understanding Application Placement
2.2.1. System Interconnnect Features Impacting Application Placement
2.2.2. Application Placement Algorithms on Cray Systems
2.3. Gathering Application Status and Information on the Cray System
2.3.1. Using the xtnodestat Command
2.4. Using the cnselect Command
2.5. Understanding How Much Memory is Available to Applications
2.6. Core Specialization
2.7. Launching an MPMD Application
2.8. Managing Compute Node Processors from an MPI Program
2.9. About aprun Input and Output Modes
2.10. About aprun Resource Limits
2.11. About aprun Signal Processing
2.12. Reserved File Descriptors
3. Running User Programs on Service Nodes
4. Using Workload Management Systems
4.1. Creating Job Scripts
4.2. Submitting Batch Jobs
4.3. Getting Job Status
4.4. Removing a Job from the Queue
5. Dynamic Shared Objects and Libraries (DSLs)
5.1. Introduction
5.2. About the Compute Node Root Run Time Environment
5.2.1. DSL Support
5.3. Configuring DSL
5.4. Building, Launching, and Workload Management Using Dynamic Objects
5.4.1. Linker Search Order
5.5. Troubleshooting
5.5.1. Error While Launching with aprun: "error while loading shared libraries"
5.5.2. Running an Application Using a Non-existent Root
5.5.3. Performance Implications of Using Dynamic Shared Objects
6. Using Cluster Compatibility Mode in CLE
6.1. Cluster Compatibility Mode
6.1.1. CCM Implementation
6.2. Installation and Configuration of Applications for CCM
6.3. Using CCM
6.3.1. CCM Commands ccmrun ccmlogin
6.3.2. Starting a CCM Batch Job
6.3.3. X11 Forwarding in CCM
6.3.4. ISV Application Acceleration (IAA) Configuring Platform MPI (HP-MPI) and Launching mpirun Caveats and Limitations for IAA Troubleshooting IAA
6.4. Individual Software Vendor (ISV) Example
6.5. Troubleshooting
6.5.1. CCM Initialization Fails
6.5.2. Is Incompatible with CCM
6.5.4. Job Hangs When sa Parameter Is Passed to Platform MPI
6.5.5. "MPI_Init: dlopen" Error(s)
6.5.6. Bus Errors In an Application, MPI, or libibgni
6.5.7. Errors at Start of Application Launch
6.5.8. "orted: command not found"
6.6. Caveats and Limitations for CCM
6.6.1. ALPS Does Not Accurately Reflect CCM Job Resources
6.6.2. Open MPI and Moab and TORQUE Integration Not Supported
6.6.3. Miscellaneous Limitations
7. Using Checkpoint/Restart
8. Optimizing Applications
8.1. Using Compiler Optimization Options
8.2. Using aprun Memory Affinity Options
8.3. Using aprun CPU Affinity Options
8.4. Exclusive Access
8.5. Optimizing Process Placement on Multicore Nodes
9. Example Applications
9.1. Running a Basic Application
9.2. Running an MPI Application
9.3. Using the Cray shmem_put Function
9.4. Using the Cray shmem_get Function
9.5. Running Partitioned Global Address Space (PGAS) Applications
9.5.1. Running a Unified Parallel C (UPC) Application
9.5.2. Running a Fortran 2008 Application Using Coarrays
9.6. Running an Acclerated Cray LibSci Routine
9.7. Running a PETSc Application
9.8. Running an OpenMP Application
9.9. Running an Interactive Batch Job
9.10. Running a Batch Job Script
9.11. Running Multiple Sequential Applications
9.12. Running Multiple Parallel Applications
9.13. Using aprun Memory Affinity Options
9.13.1. Using the aprun -S Option
9.13.2. Using the aprun -sl Option
9.13.3. Using the aprun -sn Option
9.13.4. Using the aprun -ss Option
9.14. Using aprun CPU Affinity Options
9.14.1. Using the aprun -cc cpu_list Option
9.14.2. Using the aprun -cc keyword Options
9.15. Using Checkpoint/Restart Commands
9.16. Running Compute Node Commands
9.17. Using the High-level PAPI Interface
9.18. Using the Low-level PAPI Interface
9.19. Using CrayPat
9.20. Using Cray Apprentice2
A. Further Information
A.1. Related Publications
A.1.1. Publications for Application Developers

List of Figures

1. Cabinet View Showing Three Applications in Original Serial Ordering
2. Cabinet View Showing Three Applications in New Ordering
3. Topology View of Original Application Ordering
4. Topology View of New Application Ordering
5. SeaStar Interconnect Links
6. Gemini Interconnect Links
7. Cray Job Distribution Cross Section
8. CCM Job Flow Diagram
9. Cray Apprentice2 Callgraph

List of Tables

1. aprun Versus qsub Versus bsub (LSF) Options

List of Examples

1. Compiling an application
2. Running an application in interactive mode
3. Running an application using a workload management system
4. Running a program using a batch script
5. Launching a CCM application using PBS or Moab and TORQUE
6. Launching a CCM application using Platform LSF
7. Launching the UMT/pyMPI benchmark using CCM

List of Procedures

1. Disabling CSA Accounting for the cnos class view