SHMEM

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

SHMEM (from Cray Research's “shared memory” library[1]) is a family of parallel programming libraries, providing one-sided, RDMA, parallel-processing interfaces for low-latency distributed-memory supercomputers. The SHMEM acronym was subsequently reverse engineered to mean "Symmetric Hierarchical MEMory”.[2] Later it was expanded to distributed memory parallel computer clusters, and is used as parallel programming interface or as low-level interface to build partitioned global address space (PGAS) systems and languages.[3] “Libsma”, the first SHMEM library, was created by Richard Smith at Cray Research in 1993 as a set of thin interfaces to access the CRAY T3D's inter-processor-communication hardware. SHMEM has been implemented by Cray Research, SGI, Cray Inc., Quadrics, HP, GSHMEM, IBM, QLogic, Mellanox, Universities of Houston and Florida; there is also open-source OpenSHMEM.[4]

SHMEM laid the foundations for low-latency (sub-microsecond) one-sided communication.[5] After its use on the CRAY T3E,[6] its popularity waned as few machines could deliver the near-microsecond latencies necessary to maintain efficiency for its hallmark individual-word communication. With the advent of popular sub-microsecond interconnects, SHMEM has been used to address the necessity of hyper-efficient, portable, parallel-communication methods for exascale computing.[7]

Programs written using SHMEM can be started on several computers, connected together with some high-performance network, supported by used SHMEM library. Every computer runs a copy of a program (SPMD); each copy is called PE (processing element). PEs can ask the SHMEM library to do remote memory-access operations, like reading ("shmem_get" operation) or writing ("shmem_put" operation) data. Peer-to-peer operations are one-sided, which means that no active cooperation from remote thread is needed to complete the action (but it can poll its local memory for changes using "shmem_wait"). Operations can be done on short types like bytes or words, or on longer datatypes like arrays, sometimes evenly strided or indexed (only some elements of array are sent). For short datatypes, SHMEM can do atomic operations (CAS, fetch and add, atomic increment, etc.) even in remote memory. Also there are two different synchronization methods:[4] task control sync (barriers and locks) and functions to enforce memory fencing and ordering. SHMEM has several collective operations, which should be started by all PEs, like reductions, broadcast, collect.

Every PE has some of its memory declared as "symmetric" segment (or shared memory area) and other memory is private. Only "shared" memory can be accessed in one-sided operation from remote PEs. Programmers can use static-memory constructs or shmalloc/shfree routines to create objects with symmetric address that span the PEs.

Typical SHMEM functions

[edit | edit source]
  • start_pes(N) — start N processing elements (PE)
  • _my_pe() — ask SHMEM to return the PE identifier of current thread
  • shmem_barrier_all() — wait until all PEs run up to barrier; then enable them to go further
  • shmem_put(target, source, length, pe) — write data of length "length" to the remote address "target" on PE with id "pe" from local address "source"
  • shmem_get(target, source, length, pe) — read data of length "length" from the remote address "source" on PE with id "pe" and save to read values into local address "target"[8]

List of SHMEM implementations

[edit | edit source]
  • Cray Research: Original SHMEM for Cray T3D, Cray T3E, and Cray Research PVP supercomputers[9]
  • SGI: SGI-SHMEM for systems with NUMAlink and Altix build with InfiniBand network adapters
  • Cray Inc.: MP-SHMEM for Unicos MP (X1E supercomputer)
  • Cray Inc.: LC-SHMEM for Unicos LC (Cray XT3, XT4, XT5)
  • Quadrics: Q-SHMEM[10] for Linux clusters with QsNet interconnect[9]
  • Cyclops-64 SHMEM
  • HP SHMEM[9]
  • IBM SHMEM[9]
  • GPSHMEM[9]
  • NVSHMEM is based on the OpenSHMEM API and supports GPUs, including both host- and device-initiated communication.

OpenSHMEM implementations

[edit | edit source]

OpenSHMEM is a standard effort by SGI and Open Source Software Solutions, Inc.

  • University of Houston: Reference OpenSHMEM[4][9]
  • Mellanox ScalableSHMEM[9]
  • Portals-SHMEM (on top of Portals interface)
  • University of Florida: Gator SHMEM[9]
  • Open MPI includes an implementation of OpenSHMEM[11]
  • Adapteva Epiphany Coprocessor[12]
  • Sandia OpenSHMEM (SOS) supports multiple networking APIs [13]
  • OpenSHMEM over MPI (OSHMPI) version 1 is based on MPI one-sided communication and supports the OpenSHMEM 1.2 specification[14]
  • OpenSHMEM over MPI (OSHMPI) version 2 is based on MPI one-sided communication and supports the OpenSHMEM 1.4 specification[15]
  • MVAPICH2-X includes an OpenSHMEM implementation
  • shmem4py provides Python support for the OpenSHMEM 1.5 specification functionality, as a wrapper around a C implementation.[16]

Disadvantages

[edit | edit source]

In first years SHMEM was accessible only on some Cray Research machines (later additionally on SGI)[1] equipped with special networks, limiting library widespread and being vendor lock-in (for example, Cray Research recommended to partially rewrite MPI programs to combine both MPI and shmem calls, which make the program non-portable to other clear-MPI environment).

SHMEM was not defined as standard,[9][1] so there were created several incompatible variants of SHMEM libraries by other vendors. Libraries had different include file names, different management function names for starting PEs or getting current PE id,[9] and some functions were changed or not supported.

Variants of SHMEM libraries can run on top of any MPI library, even when a cluster has only non-RDMA optimized Ethernet, however the performance will be typically worse than other enhanced networking protocols.

Memory in shared region should be allocated using special functions (shmalloc/shfree), not with the system malloc.[9]

Many disadvantages of SHMEM have been overcome with the use of OpenSHMEM on popular sub-microsecond interconnects, driven by exascale development.[7]

See also

[edit | edit source]

References

[edit | edit source]
  1. ^ a b c Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
  2. ^ Introduction to Parallel Computing - 3.11 Related Work // cse590o course, University of Washington, Winter 2002; page 154
  3. ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
  4. ^ a b c Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
  5. ^ Tools for Benchmarking, Tracing, and Simulating SHMEM Applications // CUG 2012, paper by San Diego Supercomputer center and ORNL
  6. ^ Recent Advances in Parallel Virtual Machine and Message Passing ..., Volume 11 page 59: "One-sided communication as a programming paradigm was made popular initially by the SHMEM library on the Cray T3D and T3E..."
  7. ^ a b Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
  8. ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
  9. ^ a b c d e f g h i j k Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
  10. ^ Shmem Programming Manual // Quadrics, 2000-2001
  11. ^ OpenMPI
  12. ^ James Ross and David Richie. An OpenSHMEM Implementation for the Adapteva Epiphany Coprocessor. In Proceedings of the Third Workshop on OpenSHMEM and Related Technologies, Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).. Springer, 2016.
  13. ^ Sandia OpenSHMEM (SOS) on Github
  14. ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
  15. ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).
  16. ^ Lua error in Module:Citation/CS1/Configuration at line 2172: attempt to index field '?' (a nil value).

Further reading

[edit | edit source]
[edit | edit source]
Retrieved from "http://70.231.62.181/index.php?title=SHMEM&oldid=18443240"