# Single-node Computation

## Introduction

In this chapter, all base objects (matrices, vectors and stencils) for computation on a single-node (shared-memory) system are described. A typical configuration is illustrated in Fig. 4.

Fig. 4 A typical single-node configuration, where gray boxes represent the cores, blue boxes represent the memory and arrows represent the bandwidth.

The compute node contains none, one or more accelerators. The compute node could be any kind of shared-memory (single, dual, quad CPU) system.

Note

The host and accelerator memory can be physically different.

## ValueType

The value (data) type of the vectors and the matrices is defined as a template. The matrix can be of type float (32-bit), double (64-bit) and complex (64/128-bit). The vector can be float (32-bit), double (64-bit), complex (64/128-bit) and int (32/64-bit). The information about the precision of the data type is shown in the `rocalution::BaseRocalution::Info()` function.

## Complex Support

Currently, rocALUTION does not support complex computation.

## Allocation and Free

void rocalution::LocalVector::Allocate(std::string name, IndexType2 size)

  Allocate a local vector with name and size.

  The local vector allocation function requires a name of the object (this is only for information purposes) and corresponding size description for vector objects.

  Example

        LocalVector<ValueType> vec;

        vec.Allocate("my vector", 100);
        vec.Clear();

  Parameters:

    - name – [in] object name

    - size – [in] number of elements in the vector

virtual void rocalution::LocalVector::Clear()

  Clear (free all data) the object.

void rocalution::LocalMatrix::AllocateCOO(const std::string &name, int nnz, int nrow, int ncol)

void rocalution::LocalMatrix::AllocateCSR(const std::string &name, int nnz, int nrow, int ncol)

void rocalution::LocalMatrix::AllocateBCSR(const std::string &name, int nnzb, int nrowb, int ncolb, int blockdim)

void rocalution::LocalMatrix::AllocateMCSR(const std::string &name, int nnz, int nrow, int ncol)

void rocalution::LocalMatrix::AllocateELL(const std::string &name, int nnz, int nrow, int ncol, int max_row)

void rocalution::LocalMatrix::AllocateDIA(const std::string &name, int nnz, int nrow, int ncol, int ndiag)

void rocalution::LocalMatrix::AllocateHYB(const std::string &name, int ell_nnz, int coo_nnz, int ell_max_row, int nrow, int ncol)

void rocalution::LocalMatrix::AllocateDENSE(const std::string &name, int nrow, int ncol)

  Allocate a local matrix with name and sizes.

  The local matrix allocation functions require a name of the object (this is only for information purposes) and corresponding number of non-zero elements, number of rows and number of columns. Furthermore, depending on the matrix format, additional parameters are required.

  Example

        LocalMatrix<ValueType> mat;

        mat.AllocateCSR("my CSR matrix", 456, 100, 100);
        mat.Clear();

        mat.AllocateCOO("my COO matrix", 200, 100, 100);
        mat.Clear();

Note

More detailed information on the additional parameters required for matrix allocation is given in Matrix Formats.

virtual void rocalution::LocalMatrix::Clear(void)

  Clear (free all data) the object.

## Matrix Formats

Matrices, where most of the elements are equal to zero, are called sparse. In most practical applications, the number of non-zero entries is proportional to the size of the matrix (e.g. typically, if the matrix \(A \in \mathbb{R}^{N \times N}\), then the number of elements are of order \(O(N)\)). To save memory, storing zero entries can be avoided by introducing a structure corresponding to the non-zero elements of the matrix. rocALUTION supports sparse CSR, MCSR, COO, ELL, DIA, HYB and dense matrices (DENSE).

Note

The functionality of every matrix object is different and depends on the matrix format. The CSR format provides the highest support for various functions. For a few operations, an internal conversion is performed, however, for many routines an error message is printed and the program is terminated.

Note

In the current version, some of the conversions are performed on the host (disregarding the actual object allocation - host or accelerator).

    // Convert mat to CSR storage format
    mat.ConvertToCSR();
    // Perform a matrix-vector multiplication y = mat * x in CSR format
    mat.Apply(x, &y);

    // Convert mat to ELL storage format
    mat.ConvertToELL();
    // Perform a matrix-vector multiplication y = mat * x in ELL format
    mat.Apply(x, &y);

    // Convert mat to CSR storage format
    mat.ConvertTo(CSR);
    // Perform a matrix-vector multiplication y = mat * x in CSR format
    mat.Apply(x, &y);

    // Convert mat to ELL storage format
    mat.ConvertTo(ELL);
    // Perform a matrix-vector multiplication y = mat * x in ELL format
    mat.Apply(x, &y);

### COO storage format

The most intuitive sparse format is the coordinate format (COO). It represents the non-zero elements of the matrix by their coordinates and requires two index arrays (one for row and one for column indexing) and the values array. A \(m \times n\) matrix is represented by

m | number of rows (integer).
n | number of columns (integer).
nnz | number of non-zero elements (integer).
coo_val | array of `nnz` elements containing the data (floating point).
coo_row_ind | array of `nnz` elements containing the row indices (integer).
coo_col_ind | array of `nnz` elements containing the column indices (integer).

Note

The COO matrix is expected to be sorted by row indices and column indices per row. Furthermore, each pair of indices should appear only once.

Consider the following \(3 \times 5\) matrix and the corresponding COO structures, with \(m = 3, n = 5\) and \(\text{nnz} = 8\):

\[\begin{split}A = \begin{pmatrix} 1.0 & 2.0 & 0.0 & 3.0 & 0.0 \\ 0.0 & 4.0 & 5.0 & 0.0 & 0.0 \\ 6.0 & 0.0 & 0.0 & 7.0 & 8.0 \\ \end{pmatrix}\end{split}\]

where

\[\begin{split}\begin{array}{ll} \text{coo_val}[8] & = \{1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0\} \\ \text{coo_row_ind}[8] & = \{0, 0, 0, 1, 1, 2, 2, 2\} \\ \text{coo_col_ind}[8] & = \{0, 1, 3, 1, 2, 0, 3, 4\} \end{array}\end{split}\]

### CSR storage format

One of the most popular formats in many scientific codes is the compressed sparse row (CSR) format. In this format, instead of row indices, the row offsets to the beginning of each row are stored. Thus, each row elements can be accessed sequentially. However, this format does not allow sequential accessing of the column entries. The CSR storage format represents a \(m \times n\) matrix by

m | number of rows (integer).
n | number of columns (integer).
nnz | number of non-zero elements (integer).
csr_val | array of `nnz` elements containing the data (floating point).
csr_row_ptr | array of `m+1` elements that point to the start of every row (integer).
csr_col_ind | array of `nnz` elements containing the column indices (integer).

Note

The CSR matrix is expected to be sorted by column indices within each row. Furthermore, each pair of indices should appear only once.

Consider the following \(3 \times 5\) matrix and the corresponding CSR structures, with \(m = 3, n = 5\) and \(\text{nnz} = 8\):

\[\begin{split}A = \begin{pmatrix} 1.0 & 2.0 & 0.0 & 3.0 & 0.0 \\ 0.0 & 4.0 & 5.0 & 0.0 & 0.0 \\ 6.0 & 0.0 & 0.0 & 7.0 & 8.0 \\ \end{pmatrix}\end{split}\]

where

\[\begin{split}\begin{array}{ll} \text{csr_val}[8] & = \{1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0\} \\ \text{csr_row_ptr}[4] & = \{0, 3, 5, 8\} \\ \text{csr_col_ind}[8] & = \{0, 1, 3, 1, 2, 0, 3, 4\} \end{array}\end{split}\]

### BCSR storage format

The Block Compressed Sparse Row (BCSR) storage format represents a \((mb \cdot \text{bcsr_dim}) \times (nb \cdot \text{bcsr_dim})\) matrix by

mb | number of block rows (integer)
nb | number of block columns (integer)
nnzb | number of non-zero blocks (integer)
bcsr_val | array of `nnzb * bcsr_dim * bcsr_dim` elements containing the data (floating point). Data within each block is stored in column-major.
bcsr_row_ptr | array of `mb+1` elements that point to the start of every block row (integer).
bcsr_col_ind | array of `nnzb` elements containing the block column indices (integer).
bcsr_dim | dimension of each block (integer).

The BCSR matrix is expected to be sorted by column indices within each row. If \(m\) or \(n\) are not evenly divisible by the block dimension, then zeros are padded to the matrix, such that \(mb = (m + \text{bcsr_dim} - 1) / \text{bcsr_dim}\) and \(nb = (n + \text{bcsr_dim} - 1) / \text{bcsr_dim}\). Consider the following \(4 \times 3\) matrix and the corresponding BCSR structures, with \(\text{bcsr_dim} = 2, mb = 2, nb = 2\) and \(\text{nnzb} = 4\) using zero based indexing and column-major storage:

\[\begin{split}A = \begin{pmatrix} 1.0 & 0.0 & 2.0 \\ 3.0 & 0.0 & 4.0 \\ 5.0 & 6.0 & 0.0 \\ 7.0 & 0.0 & 8.0 \\ \end{pmatrix}\end{split}\]

with the blocks \(A_{ij}\)

\[\begin{split}A_{00} = \begin{pmatrix} 1.0 & 0.0 \\ 3.0 & 0.0 \\ \end{pmatrix}, A_{01} = \begin{pmatrix} 2.0 & 0.0 \\ 4.0 & 0.0 \\ \end{pmatrix}, A_{10} = \begin{pmatrix} 5.0 & 6.0 \\ 7.0 & 0.0 \\ \end{pmatrix}, A_{11} = \begin{pmatrix} 0.0 & 0.0 \\ 8.0 & 0.0 \\ \end{pmatrix}\end{split}\]

such that

\[\begin{split}A = \begin{pmatrix} A_{00} & A_{01} \\ A_{10} & A_{11} \\ \end{pmatrix}\end{split}\]

with arrays representation

\[\begin{split}\begin{array}{ll} \text{bcsr_val}[16] & = \{1.0, 3.0, 0.0, 0.0, 2.0, 4.0, 0.0, 0.0, 5.0, 7.0, 6.0, 0.0, 0.0, 8.0, 0.0, 0.0\} \\ \text{bcsr_row_ptr}[3] & = \{0, 2, 4\} \\ \text{bcsr_col_ind}[4] & = \{0, 1, 0, 1\} \end{array}\end{split}\]

### ELL storage format

The Ellpack-Itpack (ELL) storage format can be seen as a modification of the CSR format without row offset pointers. Instead, a fixed number of elements per row is stored. It represents a \(m \times n\) matrix by

m | number of rows (integer).
n | number of columns (integer).
ell_width | maximum number of non-zero elements per row (integer)
ell_val | array of `m times ell_width` elements containing the data (floating point).
ell_col_ind | array of `m times ell_width` elements containing the column indices (integer).

Note

The ELL matrix is assumed to be stored in column-major format. Rows with less than `ell_width` non-zero elements are padded with zeros (`ell_val`) and \(-1\) (`ell_col_ind`).

Consider the following \(3 \times 5\) matrix and the corresponding ELL structures, with \(m = 3, n = 5\) and \(\text{ell_width} = 3\):

\[\begin{split}A = \begin{pmatrix} 1.0 & 2.0 & 0.0 & 3.0 & 0.0 \\ 0.0 & 4.0 & 5.0 & 0.0 & 0.0 \\ 6.0 & 0.0 & 0.0 & 7.0 & 8.0 \\ \end{pmatrix}\end{split}\]

where

\[\begin{split}\begin{array}{ll} \text{ell_val}[9] & = \{1.0, 4.0, 6.0, 2.0, 5.0, 7.0, 3.0, 0.0, 8.0\} \\ \text{ell_col_ind}[9] & = \{0, 1, 0, 1, 2, 3, 3, -1, 4\} \end{array}\end{split}\]

### DIA storage format

If all (or most) of the non-zero entries belong to a few diagonals of the matrix, they can be stored with the corresponding offsets. The values in DIA format are stored as array with size \(D \times N_D\), where \(D\) is the number of diagonals in the matrix and \(N_D\) is the number of elements in the main diagonal. Since not all values in this array are occupied, the not accessible entries are denoted with \(\ast\). They correspond to the offsets in the diagonal array (negative values represent offsets from the beginning of the array). The DIA storage format represents a \(m \times n\) matrix by

m | number of rows (integer)
n | number of columns (integer)
ndiag | number of occupied diagonals (integer)
dia_offset | array of `ndiag` elements containing the offset with respect to the main diagonal (integer).
dia_val | array of `m times ndiag` elements containing the values (floating point).

Consider the following \(5 \times 5\) matrix and the corresponding DIA structures, with \(m = 5, n = 5\) and \(\text{ndiag} = 4\):

\[\begin{split}A = \begin{pmatrix} 1 & 2 & 0 & 11 & 0 \\ 0 & 3 & 4 & 0 & 0 \\ 0 & 5 & 6 & 7 & 0 \\ 0 & 0 & 0 & 8 & 0 \\ 0 & 0 & 0 & 9 & 10 \end{pmatrix}\end{split}\]

where

\[\begin{split}\begin{array}{ll} \text{dia_val}[20] & = \{\ast, 0, 5, 0, 9, 1, 3, 6, 8, 10, 2, 4, 7, 0, \ast, 11, 0, \ast, \ast, \ast\} \\ \text{dia_offset}[4] & = \{-1, 0, 1, 3\} \end{array}\end{split}\]

### HYB storage format

The DIA and ELL formats cannot represent efficiently completely unstructured sparse matrices. To keep the memory footprint low, DIA requires the elements to belong to a few diagonals and ELL needs a fixed number of elements per row. For many applications this is a too strong restriction. A solution to this issue is to represent the more regular part of the matrix in such a format and the remaining part in COO format. The HYB format is a mixture between ELL and COO, where the maximum elements per row for the ELL part is computed by nnz/m. It represents a \(m \times n\) matrix by

m | number of rows (integer).
n | number of columns (integer).
nnz | number of non-zero elements of the COO part (integer)
ell_width | maximum number of non-zero elements per row of the ELL part (integer)
ell_val | array of `m times ell_width` elements containing the ELL part data (floating point).
ell_col_ind | array of `m times ell_width` elements containing the ELL part column indices (integer).
coo_val | array of `nnz` elements containing the COO part data (floating point).
coo_row_ind | array of `nnz` elements containing the COO part row indices (integer).
coo_col_ind | array of `nnz` elements containing the COO part column indices (integer).

### Memory Usage

The memory footprint of the different matrix formats is presented in the following table, considering a \(N \times N\) matrix, where the number of non-zero entries is denoted with nnz.

Format | Structure | Values
--- | --- | ---
DENSE |  | \(N \times N\)
COO | \(2 \times \text{nnz}\) | \(\text{nnz}\)
CSR | \(N + 1 + \text{nnz}\) | \(\text{nnz}\)
ELL | \(M \times N\) | \(M \times N\)
DIA | \(D\) | \(D \times N_D\)

For the ELL matrix \(M\) characterizes the maximal number of non-zero elements per row and for the DIA matrix, \(D\) defines the number of diagonals and \(N_D\) defines the size of the main diagonal.

## File I/O

virtual void rocalution::LocalVector::ReadFileASCII(const std::string &filename)

  Read vector from ASCII file.

  Read a vector from ASCII file.

  Example

        LocalVector<ValueType> vec;
        vec.ReadFileASCII("my_vector.dat");

  Parameters:

    filename – [in] name of the file containing the ASCII data.

virtual void rocalution::LocalVector::WriteFileASCII(const std::string &filename) const

  Write vector to ASCII file.

  Write a vector to ASCII file.

  Example

        LocalVector<ValueType> vec;

        // Allocate and fill vec
        // ...

        vec.WriteFileASCII("my_vector.dat");

  Parameters:

    filename – [in] name of the file to write the ASCII data to.

virtual void rocalution::LocalVector::ReadFileBinary(const std::string &filename)

  Read vector from binary file.

  Read a vector from binary file. For details on the format, see WriteFileBinary().

  Example

        LocalVector<ValueType> vec;
        vec.ReadFileBinary("my_vector.bin");

  Parameters:

    filename – [in] name of the file containing the data.

virtual void rocalution::LocalVector::WriteFileBinary(const std::string &filename) const

  Write vector to binary file.

  Write a vector to binary file.

  The binary format contains a header, the rocALUTION version and the vector data as follows

      // Header
      out << "#rocALUTION binary vector file" << std::endl;

      // rocALUTION version
      out.write((char*)&version, sizeof(int));

      // Vector data
      out.write((char*)&size, sizeof(int));
      out.write((char*)vec_val, size * sizeof(double));

  Example

        LocalVector<ValueType> vec;

        // Allocate and fill vec
        // ...

        vec.WriteFileBinary("my_vector.bin");

  Note

  Vector values array is always stored in double precision (e.g. double or std::complex<double>).

  Parameters:

    filename – [in] name of the file to write the data to.

void rocalution::LocalMatrix::ReadFileMTX(const std::string &filename)

  Read matrix from MTX (Matrix Market Format) file.

  Read a matrix from Matrix Market Format file.

  Example

        LocalMatrix<ValueType> mat;
        mat.ReadFileMTX("my_matrix.mtx");

  Parameters:

    filename – [in] name of the file containing the MTX data.

void rocalution::LocalMatrix::WriteFileMTX(const std::string &filename) const

  Write matrix to MTX (Matrix Market Format) file.

  Write a matrix to Matrix Market Format file.

  Example

        LocalMatrix<ValueType> mat;

        // Allocate and fill mat
        // ...

        mat.WriteFileMTX("my_matrix.mtx");

  Parameters:

    filename – [in] name of the file to write the MTX data to.

void rocalution::LocalMatrix::ReadFileCSR(const std::string &filename)

  Read matrix from CSR (rocALUTION binary format) file.

  Read a CSR matrix from binary file. For details on the format, see WriteFileCSR().

  Example

        LocalMatrix<ValueType> mat;
        mat.ReadFileCSR("my_matrix.csr");

  Parameters:

    filename – [in] name of the file containing the data.

void rocalution::LocalMatrix::WriteFileCSR(const std::string &filename) const

  Write CSR matrix to binary file.

  Write a CSR matrix to binary file.

  The binary format contains a header, the rocALUTION version and the matrix data as follows

      // Header
      out << "#rocALUTION binary csr file" << std::endl;

      // rocALUTION version
      out.write((char*)&version, sizeof(int));

      // CSR matrix data
      out.write((char*)&m, sizeof(int));
      out.write((char*)&n, sizeof(int));
      out.write((char*)&nnz, sizeof(int));
      out.write((char*)csr_row_ptr, (m + 1) * sizeof(int));
      out.write((char*)csr_col_ind, nnz * sizeof(int));
      out.write((char*)csr_val, nnz * sizeof(double));

  Example

        LocalMatrix<ValueType> mat;

        // Allocate and fill mat
        // ...

        mat.WriteFileCSR("my_matrix.csr");

  Note

  Vector values array is always stored in double precision (e.g. double or std::complex<double>).

  Parameters:

    filename – [in] name of the file to write the data to.

## Access

ValueType &rocalution::LocalVector::operator[](int i)

const ValueType &rocalution::LocalVector::operator[](int i) const

  Access operator (only for host data)

  The elements in the vector can be accessed via [] operators, when the vector is allocated on the host.

  Example

        // rocALUTION local vector object
        LocalVector<ValueType> vec;

        // Allocate vector
        vec.Allocate("my_vector", 100);

        // Initialize vector with 1
        vec.Ones();

        // Set even elements to -1
        for(int i = 0; i < vec.GetSize(); i += 2)
        {
          vec[i] = -1;
        }

  Parameters:

    i – [in] access data at index `i`

  Returns:

    value at index `i`

Note

Accessing elements via the [] operators is slow. Use this for debugging purposes only. There is no direct access to the elements of matrices due to the sparsity structure. Matrices can be imported by a copy function. For CSR matrices, this is `rocalution::LocalMatrix::CopyFromCSR()` and `rocalution::LocalMatrix::CopyToCSR()`.

    // Allocate the CSR matrix
    int* csr_row_ptr   = new int[100 + 1];
    int* csr_col_ind   = new int[345];
    ValueType* csr_val = new ValueType[345];

    // Fill the CSR matrix
    // ...

    // rocALUTION local matrix object
    LocalMatrix<ValueType> mat;

    // Import CSR matrix to rocALUTION
    mat.AllocateCSR("my_matrix", 345, 100, 100);
    mat.CopyFromCSR(csr_row_ptr, csr_col, csr_val);

## Raw Access to the Data

### SetDataPtr

For vector and matrix objects, direct access to the raw data can be obtained via pointers. Already allocated data can be set with SetDataPtr. Setting data pointers will leave the original pointers empty.

void rocalution::LocalVector::SetDataPtr(ValueType **ptr, std::string name, int size)

  Initialize a LocalVector on the host with externally allocated data.

  `SetDataPtr` has direct access to the raw data via pointers. Already allocated data can be set by passing the pointer.

  Example

        // Allocate vector
        ValueType* ptr_vec = new ValueType[200];

        // Fill vector
        // ...

        // rocALUTION local vector object
        LocalVector<ValueType> vec;

        // Set the vector data, ptr_vec will become invalid
        vec.SetDataPtr(&ptr_vec, "my_vector", 200);

  Note

  Setting data pointer will leave the original pointer empty (set to `NULL`).

void rocalution::LocalMatrix::SetDataPtrCOO(int **row, int **col, ValueType **val, std::string name, int nnz, int nrow, int ncol)

void rocalution::LocalMatrix::SetDataPtrCSR(int **row_offset, int **col, ValueType **val, std::string name, int nnz, int nrow, int ncol)

void rocalution::LocalMatrix::SetDataPtrMCSR(int **row_offset, int **col, ValueType **val, std::string name, int nnz, int nrow, int ncol)

void rocalution::LocalMatrix::SetDataPtrELL(int **col, ValueType **val, std::string name, int nnz, int nrow, int ncol, int max_row)

void rocalution::LocalMatrix::SetDataPtrDIA(int **offset, ValueType **val, std::string name, int nnz, int nrow, int ncol, int num_diag)

void rocalution::LocalMatrix::SetDataPtrDENSE(ValueType **val, std::string name, int nrow, int ncol)

  Initialize a LocalMatrix on the host with externally allocated data.

  `SetDataPtr` functions have direct access to the raw data via pointers. Already allocated data can be set by passing their pointers.

  Example

        // Allocate a CSR matrix
        int* csr_row_ptr   = new int[100 + 1];
        int* csr_col_ind   = new int[345];
        ValueType* csr_val = new ValueType[345];

        // Fill the CSR matrix
        // ...

        // rocALUTION local matrix object
        LocalMatrix<ValueType> mat;

        // Set the CSR matrix data, csr_row_ptr, csr_col and csr_val pointers become
        // invalid
        mat.SetDataPtrCSR(&csr_row_ptr, &csr_col, &csr_val, "my_matrix", 345, 100, 100);

  Note

  Setting data pointers will leave the original pointers empty (set to `NULL`).

### LeaveDataPtr

With LeaveDataPtr, the raw data from the object can be obtained. This will leave the object empty.

void rocalution::LocalVector::LeaveDataPtr(ValueType **ptr)

  Leave a LocalVector to host pointers.

  `LeaveDataPtr` has direct access to the raw data via pointers. A LocalVector object can leave its raw data to a host pointer. This will leave the LocalVector empty.

  Example

        // rocALUTION local vector object
        LocalVector<ValueType> vec;

        // Allocate the vector
        vec.Allocate("my_vector", 100);

        // Fill vector
        // ...

        ValueType* ptr_vec = NULL;

        // Get (steal) the data from the vector, this will leave the local vector object empty
        vec.LeaveDataPtr(&ptr_vec);

void rocalution::LocalMatrix::LeaveDataPtrCOO(int **row, int **col, ValueType **val)

void rocalution::LocalMatrix::LeaveDataPtrCSR(int **row_offset, int **col, ValueType **val)

void rocalution::LocalMatrix::LeaveDataPtrMCSR(int **row_offset, int **col, ValueType **val)

void rocalution::LocalMatrix::LeaveDataPtrELL(int **col, ValueType **val, int &max_row)

void rocalution::LocalMatrix::LeaveDataPtrDIA(int **offset, ValueType **val, int &num_diag)

void rocalution::LocalMatrix::LeaveDataPtrDENSE(ValueType **val)

  Leave a LocalMatrix to host pointers.

  `LeaveDataPtr` functions have direct access to the raw data via pointers. A LocalMatrix object can leave its raw data to host pointers. This will leave the LocalMatrix empty.

  Example

        // rocALUTION CSR matrix object
        LocalMatrix<ValueType> mat;

        // Allocate the CSR matrix
        mat.AllocateCSR("my_matrix", 345, 100, 100);

        // Fill CSR matrix
        // ...

        int* csr_row_ptr   = NULL;
        int* csr_col_ind   = NULL;
        ValueType* csr_val = NULL;

        // Get (steal) the data from the matrix, this will leave the local matrix
        // object empty
        mat.LeaveDataPtrCSR(&csr_row_ptr, &csr_col_ind, &csr_val);

Note

If the object is allocated on the host, then the pointers obtained from SetDataPtr and LeaveDataPtr will be on the host. If the vector object is on the accelerator, then the data pointers will be on the accelerator.

Note

If the object is moved to and from the accelerator, then the original pointer will be invalid.

Note

Never rely on old pointers, hidden object movement to and from the accelerator will make them invalid.

Note

Whenever you pass or obtain pointers to/from a rocALUTION object, you need to use the same memory allocation/free functions. Please check the source code for that (for host src/utils/allocate_free.cpp and for HIP src/base/hip/hip_allocate_free.cpp)

## Copy CSR Matrix Host Data

void rocalution::LocalMatrix::CopyFromHostCSR(const int *row_offset, const int *col, const ValueType *val, const std::string &name, int nnz, int nrow, int ncol)

  Allocates and copies (imports) a host CSR matrix.

  If the CSR matrix data pointers are only accessible as constant, the user can create a LocalMatrix object and pass const CSR host pointers. The LocalMatrix will then be allocated and the data will be copied to the corresponding backend, where the original object was located at.

  Parameters:

    - row_offset – [in] CSR matrix row offset pointers.

    - col – [in] CSR matrix column indices.

    - val – [in] CSR matrix values array.

    - name – [in] Matrix object name.

    - nnz – [in] Number of non-zero elements.

    - nrow – [in] Number of rows.

    - ncol – [in] Number of columns.

## Copy Data

The user can copy data to and from a local vector by using CopyFromData() CopyToData().

void rocalution::LocalVector::CopyFromData(const ValueType *data)

  Copy (import) vector.

  Copy (import) vector data that is described in one array (values). The object data has to be allocated with Allocate(), using the corresponding size of the data, first.

  Parameters:

    data – [in] data to be imported.

void rocalution::LocalVector::CopyToData(ValueType *data) const

  Copy (export) vector.

  Copy (export) vector data that is described in one array (values). The output array has to be allocated, using the corresponding size of the data, first. Size can be obtain by GetSize().

  Parameters:

    data – [out] exported data.

## Object Info

virtual void rocalution::BaseRocalution::Info(void) const = 0

  Print object information.

  `Info` can print object information about any rocALUTION object. This information consists of object properties and backend data.

  Example

        mat.Info();
        vec.Info();

## Copy

All matrix and vector objects provide a CopyFrom() function. The destination object should have the same size or be empty. In the latter case, the object is allocated at the source platform.

virtual void rocalution::LocalVector::CopyFrom(const LocalVector<ValueType> &src)

  Copy vector from another vector.

  `CopyFrom` copies values from another vector.

  Example

        LocalVector<ValueType> vec1, vec2;

        // Allocate and initialize vec1 and vec2
        // ...

        // Move vec1 to accelerator
        // vec1.MoveToAccelerator();

        // Now, vec1 is on the accelerator (if available)
        // and vec2 is on the host

        // Copy vec1 to vec2 (or vice versa) will move data between host and
        // accelerator backend
        vec1.CopyFrom(vec2);

  Note

  This function allows cross platform copying. One of the objects could be allocated on the accelerator backend.

  Parameters:

    src – [in] Vector, where values should be copied from.

void rocalution::LocalMatrix::CopyFrom(const LocalMatrix<ValueType> &src)

  Copy matrix from another LocalMatrix.

  `CopyFrom` copies values and structure from another local matrix. Source and destination matrix should be in the same format.

  Example

        LocalMatrix<ValueType> mat1, mat2;

        // Allocate and initialize mat1 and mat2
        // ...

        // Move mat1 to accelerator
        // mat1.MoveToAccelerator();

        // Now, mat1 is on the accelerator (if available)
        // and mat2 is on the host

        // Copy mat1 to mat2 (or vice versa) will move data between host and
        // accelerator backend
        mat1.CopyFrom(mat2);

  Note

  This function allows cross platform copying. One of the objects could be allocated on the accelerator backend.

  Parameters:

    src – [in] Local matrix where values and structure should be copied from.

Note

For vectors, the user can specify source and destination offsets and thus copy only a part of the whole vector into another vector.

virtual void rocalution::LocalVector::CopyFrom(const LocalVector<ValueType> &src, int src_offset, int dst_offset, int size)

  Copy vector from another vector with offsets and size.

  `CopyFrom` copies values with specific source and destination offsets and sizes from another vector.

  Note

  This function allows cross platform copying. One of the objects could be allocated on the accelerator backend.

  Parameters:

    - src – [in] Vector, where values should be copied from.

    - src_offset – [in] source offset.

    - dst_offset – [in] destination offset.

    - size – [in] number of entries to be copied.

## Clone

The copy operators allow you to copy the values of the object to another object, without changing the backend specification of the object. In many algorithms, you might need auxiliary vectors or matrices. These objects can be cloned with CloneFrom().

### CloneFrom

virtual void rocalution::LocalVector::CloneFrom(const LocalVector<ValueType> &src)

  Clone the vector.

  `CloneFrom` clones the entire vector, with data and backend descriptor from another Vector.

  Example

        LocalVector<ValueType> vec;

        // Allocate and initialize vec (host or accelerator)
        // ...

        LocalVector<ValueType> tmp;

        // By cloning vec, tmp will have identical values and will be on the same
        // backend as vec
        tmp.CloneFrom(vec);

  Parameters:

    src – [in] Vector to clone from.

void rocalution::LocalMatrix::CloneFrom(const LocalMatrix<ValueType> &src)

  Clone the matrix.

  `CloneFrom` clones the entire matrix, including values, structure and backend descriptor from another LocalMatrix.

  Example

        LocalMatrix<ValueType> mat;

        // Allocate and initialize mat (host or accelerator)
        // ...

        LocalMatrix<ValueType> tmp;

        // By cloning mat, tmp will have identical values and structure and will be on
        // the same backend as mat
        tmp.CloneFrom(mat);

  Parameters:

    src – [in] LocalMatrix to clone from.

### CloneBackend

virtual void rocalution::BaseRocalution::CloneBackend(const BaseRocalution<ValueType> &src)

  Clone the Backend descriptor from another object.

  With `CloneBackend`, the backend can be cloned without copying any data. This is especially useful, if several objects should reside on the same backend, but keep their original data.

  Example

        LocalVector<ValueType> vec;
        LocalMatrix<ValueType> mat;

        // Allocate and initialize vec and mat
        // ...

        LocalVector<ValueType> tmp;
        // By cloning backend, tmp and vec will have the same backend as mat
        tmp.CloneBackend(mat);
        vec.CloneBackend(mat);

        // The following matrix vector multiplication will be performed on the backend
        // selected in mat
        mat.Apply(vec, &tmp);

  Parameters:

    src – [in] Object, where the backend should be cloned from.

## Check

virtual bool rocalution::LocalVector::Check(void) const

  Perform a sanity check of the vector.

  Checks, if the vector contains valid data, i.e. if the values are not infinity and not NaN (not a number).

  Return values:

    - true – if the vector is ok (empty vector is also ok).

    - false – if there is something wrong with the values.

bool rocalution::LocalMatrix::Check(void) const

  Perform a sanity check of the matrix.

  Checks, if the matrix contains valid data, i.e. if the values are not infinity and not NaN (not a number) and if the structure of the matrix is correct (e.g. indices cannot be negative, CSR and COO matrices have to be sorted, etc.).

  Return values:

    - true – if the matrix is ok (empty matrix is also ok).

    - false – if there is something wrong with the structure or values.

Checks, if the object contains valid data. For vectors, the function checks if the values are not infinity and not NaN (not a number). For matrices, this function checks the values and if the structure of the matrix is correct (e.g. indices cannot be negative, CSR and COO matrices have to be sorted, etc.).

## Sort

void rocalution::LocalMatrix::Sort(void)

  Sort the matrix indices.

  Sorts the matrix by indices.

  - For CSR matrices, column values are sorted.

  - For COO matrices, row indices are sorted.

## Keying

void rocalution::LocalMatrix::Key(long int &row_key, long int &col_key, long int &val_key) const

  Compute a unique hash key for the matrix arrays.

  Typically, it is hard to compare if two matrices have the same structure (and values). To do so, rocALUTION provides a keying function, that generates three keys, for the row index, column index and values array.

  Parameters:

    - row_key – [out] row index array key

    - col_key – [out] column index array key

    - val_key – [out] values array key

## Graph Analyzers

The following functions are available for analyzing the connectivity in graph of the underlying sparse matrix.

- (R)CMK Ordering

- Maximal Independent Set

- Multi-Coloring

- Zero Block Permutation

- Connectivity Ordering

All graph analyzing functions return a permutation vector (integer type), which is supposed to be used with the `rocalution::LocalMatrix::Permute()` and `rocalution::LocalMatrix::PermuteBackward()` functions in the matrix and vector classes.

### Cuthill-McKee Ordering

void rocalution::LocalMatrix::CMK(LocalVector<int> *permutation) const

  Create permutation vector for CMK reordering of the matrix.

  The Cuthill-McKee ordering minimize the bandwidth of a given sparse matrix.

  Example

        LocalVector<int> cmk;

        mat.CMK(&cmk);
        mat.Permute(cmk);

  Parameters:

    permutation – [out] permutation vector for CMK reordering

void rocalution::LocalMatrix::RCMK(LocalVector<int> *permutation) const

  Create permutation vector for reverse CMK reordering of the matrix.

  The Reverse Cuthill-McKee ordering minimize the bandwidth of a given sparse matrix.

  Example

        LocalVector<int> rcmk;

        mat.RCMK(&rcmk);
        mat.Permute(rcmk);

  Parameters:

    permutation – [out] permutation vector for reverse CMK reordering

### Maximal Independent Set

void rocalution::LocalMatrix::MaximalIndependentSet(int &size, LocalVector<int> *permutation) const

  Perform maximal independent set decomposition of the matrix.

  The Maximal Independent Set algorithm finds a set with maximal size, that contains elements that do not depend on other elements in this set.

  Example

        LocalVector<int> mis;
        int size;

        mat.MaximalIndependentSet(size, &mis);
        mat.Permute(mis);

  Parameters:

    - size – [out] number of independent sets

    - permutation – [out] permutation vector for maximal independent set reordering

### Multi-Coloring

void rocalution::LocalMatrix::MultiColoring(int &num_colors, int **size_colors, LocalVector<int> *permutation) const

  Perform multi-coloring decomposition of the matrix.

  The Multi-Coloring algorithm builds a permutation (coloring of the matrix) in a way such that no two adjacent nodes in the sparse matrix have the same color.

  Example

        LocalVector<int> mc;
        int num_colors;
        int* block_colors = NULL;

        mat.MultiColoring(num_colors, &block_colors, &mc);
        mat.Permute(mc);

  Parameters:

    - num_colors – [out] number of colors

    - size_colors – [out] pointer to array that holds the number of nodes for each color

    - permutation – [out] permutation vector for multi-coloring reordering

### Zero Block Permutation

void rocalution::LocalMatrix::ZeroBlockPermutation(int &size, LocalVector<int> *permutation) const

  Return a permutation for saddle-point problems (zero diagonal entries)

  For Saddle-Point problems, (i.e. matrices with zero diagonal entries), the Zero Block Permutation maps all zero-diagonal elements to the last block of the matrix.

  Example

        LocalVector<int> zbp;
        int size;

        mat.ZeroBlockPermutation(size, &zbp);
        mat.Permute(zbp);

  Parameters:

    - size – [out]

    - permutation – [out] permutation vector for zero block permutation

### Connectivity Ordering

void rocalution::LocalMatrix::ConnectivityOrder(LocalVector<int> *permutation) const

  Create permutation vector for connectivity reordering of the matrix.

  Connectivity ordering returns a permutation, that sorts the matrix by non-zero entries per row.

  Example

        LocalVector<int> conn;

        mat.ConnectivityOrder(&conn);
        mat.Permute(conn);

  Parameters:

    permutation – [out] permutation vector for connectivity reordering

## Basic Linear Algebra Operations

For a full list of functions and routines involving operators and vectors, see the API specifications.