Skip to main content
`Ctrl`+`K`

# RCCL library specification

This document provides details of the API library.

## Communicator functions

Warning

doxygenfunction: Cannot find function “ncclGetUniqueId” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclCommInitRank” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclCommInitAll” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclCommDestroy” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclCommAbort” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclCommCount” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclCommCuDevice” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclCommUserRank” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

## Collective communication operations

Collective communication operations must be called separately for each communicator in a communicator clique.

They return when operations have been enqueued on the hipstream.

Since they may perform inter-CPU synchronization, each call has to be done from a different thread or process, or need to use Group Semantics (see below).

Warning

doxygenfunction: Cannot find function “ncclReduce” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclBcast” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclBroadcast” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclAllReduce” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclReduceScatter” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclAllGather” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclSend” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclRecv” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclGather” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclScatter” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclAllToAll” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

## Group semantics

When managing multiple GPUs from a single thread, and since NCCL collective calls may perform inter-CPU synchronization, we need to “group” calls for different ranks/devices into a single call.

Grouping NCCL calls as being part of the same collective operation is done using ncclGroupStart and ncclGroupEnd. ncclGroupStart will enqueue all collective calls until the ncclGroupEnd call, which will wait for all calls to be complete. Note that for collective communication, ncclGroupEnd only guarantees that the operations are enqueued on the streams, not that the operation is effectively done.

Both collective communication and ncclCommInitRank can be used in conjunction of ncclGroupStart/ncclGroupEnd.

Warning

doxygenfunction: Cannot find function “ncclGroupStart” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclGroupEnd” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

## Library functions

Warning

doxygenfunction: Cannot find function “ncclGetVersion” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

Warning

doxygenfunction: Cannot find function “ncclGetErrorString” in doxygen xml output for project “RCCL 2.27.7 Documentation” from directory: /build/rccl-u8xwek/rccl-7.1.0/docs/doxygen/xml

## Types

There are few data structures that are internal to the library. The pointer types to these structures are given below. The user would need to use these types to create handles and pass them between different library functions.

typedef struct ncclComm *ncclComm_t

  Opaque handle to communicator.

  A communicator contains information required to facilitate collective communications calls

struct ncclUniqueId

  Opaque unique id used to initialize communicators.

  The ncclUniqueId must be passed to all participating ranks

## Enumerations

This section provides all the enumerations used.

enum ncclResult_t

  Result type.

  Return codes aside from ncclSuccess indicate that a call has failed

  Values:

  enumerator ncclSuccess

    No error

  enumerator ncclUnhandledCudaError

    Unhandled HIP error

  enumerator ncclSystemError

    Unhandled system error

  enumerator ncclInternalError

    Internal Error - Please report to RCCL developers

  enumerator ncclInvalidArgument

    Invalid argument

  enumerator ncclInvalidUsage

    Invalid usage

  enumerator ncclRemoteError

    Remote process exited or there was a network error

  enumerator ncclInProgress

    RCCL operation in progress

  enumerator ncclNumResults

    Number of result types

enum ncclRedOp_t

  Reduction operation selector.

  Enumeration used to specify the various reduction operations ncclNumOps is the number of built-in ncclRedOp_t values and serves as the least possible value for dynamic ncclRedOp_t values constructed by ncclRedOpCreate functions.

  ncclMaxRedOp is the largest valid value for ncclRedOp_t and is defined to be the largest signed value (since compilers are permitted to use signed enums) that won’t grow sizeof(ncclRedOp_t) when compared to previous RCCL versions to maintain ABI compatibility.

  Values:

  enumerator ncclSum

    Sum

  enumerator ncclProd

    Product

  enumerator ncclMax

    Max

  enumerator ncclMin

    Min

  enumerator ncclAvg

    Average

  enumerator ncclNumOps

    Number of built-in reduction ops

  enumerator ncclMaxRedOp

    Largest value for ncclRedOp_t

enum ncclDataType_t

  Data types.

  Enumeration of the various supported datatype

  Values:

  enumerator ncclInt8
  enumerator ncclChar
  enumerator ncclUint8
  enumerator ncclInt32
  enumerator ncclInt
  enumerator ncclUint32
  enumerator ncclInt64
  enumerator ncclUint64
  enumerator ncclFloat16
  enumerator ncclHalf
  enumerator ncclFloat32
  enumerator ncclFloat
  enumerator ncclFloat64
  enumerator ncclDouble
  enumerator ncclBfloat16
  enumerator ncclFloat8e4m3
  enumerator ncclFloat8e5m2
  enumerator ncclNumTypes