chirp_matrix.h File Reference

Stores very large distributed matrices. More...

`#include "chirp_types.h"`

Go to the source code of this file.

## Functions

struct chirp_matrix * | chirp_matrix_create (const char *host, const char *path, int width, int height, int element_size, int nhosts, time_t stoptime)
| Create a new distributed matrix.
struct chirp_matrix * | chirp_matrix_open (const char *host, const char *path, time_t stoptime)
| Open an existing matrix.
int | chirp_matrix_get_row (struct chirp_matrix *matrix, int y, void *data, time_t stoptime)
| Get all values in a row.
int | chirp_matrix_set_row (struct chirp_matrix *matrix, int y, const void *data, time_t stoptime)
| Set all values in a row.
int | chirp_matrix_get_col (struct chirp_matrix *matrix, int x, void *data, time_t stoptime)
| Get all values in a column.
int | chirp_matrix_set_col (struct chirp_matrix *matrix, int x, const void *data, time_t stoptime)
| Set all values in a column.
int | chirp_matrix_get_range (struct chirp_matrix *matrix, int x, int y, int width, int height, void *data, time_t stoptime)
| Get a range of data.
int | chirp_matrix_set_range (struct chirp_matrix *matrix, int x, int y, int width, int height, const void *data, time_t stoptime)
| Set a range of data.
int | chirp_matrix_get (struct chirp_matrix *matrix, int x, int y, void *data, time_t stoptime)
| Get a single element.
int | chirp_matrix_set (struct chirp_matrix *matrix, int x, int y, const void *data, time_t stoptime)
| Set a single element.
int | chirp_matrix_setacl (const char *host, const char *path, const char *subject, const char *rights, time_t stoptime)
| Set the acls on a matrix.
int | chirp_matrix_width (struct chirp_matrix *matrix)
| Get the width of a matrix.
int | chirp_matrix_height (struct chirp_matrix *matrix)
| Get the height of a matrix.
int | chirp_matrix_element_size (struct chirp_matrix *matrix)
| Get the element size of a matrix.
int | chirp_matrix_nhosts (struct chirp_matrix *matrix)
| Get the number of hosts used by a matrix.
int | chirp_matrix_nfiles (struct chirp_matrix *matrix)
| Get the number of files used by a matrix.
void | chirp_matrix_fsync (struct chirp_matrix *matrix, time_t stoptime)
| Force all data to disk.
void | chirp_matrix_close (struct chirp_matrix *matrix, time_t stoptime)
| Close a matrix and free all related resources.
int | chirp_matrix_delete (const char *host, const char *path, time_t stoptime)
| Delete a matrix.

## Detailed Description

Stores very large distributed matrices.

This module manages the storage of very large matrices (ranging from gigabytes to terabytes) by distributing the data across multiple Chirp servers. By harnessing the aggregate memory and storage of multiple machines, the time to process data is dramatically reduced.

## Function Documentation

## chirp_matrix_create()

struct chirp_matrix * chirp_matrix_create | ( | const char * | host,
|  | const char * | path,
|  | int | width,
|  | int | height,
|  | int | element_size,
|  | int | nhosts,
|  | time_t | stoptime )

Create a new distributed matrix.

The host and path specified here will be used to store a small index file that contains the configuration of the matrix. The actual pieces of the matrix will be spread across multiple hosts. To tell Chirp where to store those pieces, you must create a hosts file that is a simple list of host names separated by newlines. Chirp will look for this file in $CHIRP_HOSTS, then in $HOME/.chirp/hosts. If neither is available, this call will fail.

Parameters

  host | The hostname and optional port of the index file.
  path | The path to the index file.
  width | The number of elements in one row.
  height | The number of elements in one column.
  element_size | The size in bytes of each element in the matrix.
  nhosts | The number of hosts on which to spread the data.
  stoptime | The absolute time at which to abort.

Returns
  On success, a pointer to a struct chirp_matrix. On failure, returns zero and sets errno appropriately.

See also
  chirp_matrix_open, chirp_matrix_delete

## chirp_matrix_open()

struct chirp_matrix * chirp_matrix_open | ( | const char * | host,
|  | const char * | path,
|  | time_t | stoptime )

Open an existing matrix.

Parameters

  host | The hostname and optional port of the index file.
  path | The path to the index file.
  stoptime | The absolute time at which to abort.

Returns
  On success, a pointer to a struct chirp_matrix. On failure, returns zero and sets errno appropriately.

See also
  chirp_matrix_close

## chirp_matrix_get_row()

int chirp_matrix_get_row | ( | struct chirp_matrix * | matrix,
|  | int | y,
|  | void * | data,
|  | time_t | stoptime )

Get all values in a row.

This is the most efficient way to access data in a matrix.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  y | The y position of the row.
  data | A pointer to a buffer where to store the data.
  stoptime | The absolute time at which to abort.

## chirp_matrix_set_row()

int chirp_matrix_set_row | ( | struct chirp_matrix * | matrix,
|  | int | y,
|  | const void * | data,
|  | time_t | stoptime )

Set all values in a row.

This is the most efficient way to access data in a matrix.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  y | The y position of the row.
  data | A pointer to a buffer containing the data to write.
  stoptime | The absolute time at which to abort.

Returns
  Greater than or equal to zero on success, negative on failure.

## chirp_matrix_get_col()

int chirp_matrix_get_col | ( | struct chirp_matrix * | matrix,
|  | int | x,
|  | void * | data,
|  | time_t | stoptime )

Get all values in a column.

Note that accessing columns is not as efficient as accessing rows. If possible, use chirp_matrix_get_row instead.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  x | The x position of the column.
  data | A pointer to a buffer where to store the data.
  stoptime | The absolute time at which to abort.

Returns
  Greater than or equal to zero on success, negative on failure.

## chirp_matrix_set_col()

int chirp_matrix_set_col | ( | struct chirp_matrix * | matrix,
|  | int | x,
|  | const void * | data,
|  | time_t | stoptime )

Set all values in a column.

Note that accessing columns is not as efficient as accessing rows. If possible, use chirp_matrix_set_row instead.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  x | The x position of the column.
  data | A pointer to a buffer containing the data to write.
  stoptime | The absolute time at which to abort.

Returns
  Greater than or equal to zero on success, negative on failure.

## chirp_matrix_get_range()

int chirp_matrix_get_range | ( | struct chirp_matrix * | matrix,
|  | int | x,
|  | int | y,
|  | int | width,
|  | int | height,
|  | void * | data,
|  | time_t | stoptime )

Get a range of data.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  x | The starting x position of the range.
  y | The starting y position of the range;
  width | The width of the range in cells.
  height | The width of the range in cells.
  data | A pointer to a buffer where to store the data.
  stoptime | The absolute time at which to abort.

Returns
  Greater than or equal to zero on success, negative on failure.

## chirp_matrix_set_range()

int chirp_matrix_set_range | ( | struct chirp_matrix * | matrix,
|  | int | x,
|  | int | y,
|  | int | width,
|  | int | height,
|  | const void * | data,
|  | time_t | stoptime )

Set a range of data.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  x | The starting x position of the range.
  y | The starting y position of the range;
  width | The width of the range in cells.
  height | The width of the range in cells.
  data | A pointer to a buffer containing the data to write.
  stoptime | The absolute time at which to abort.

Returns
  Greater than or equal to zero on success, negative on failure.

## chirp_matrix_get()

int chirp_matrix_get | ( | struct chirp_matrix * | matrix,
|  | int | x,
|  | int | y,
|  | void * | data,
|  | time_t | stoptime )

Get a single element.

Note: Reading a single element at a time is very inefficient. If possible, get multiple elements at once using chirp_matrix_get_row.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  x | The x position of the element.
  y | The y position of the element.
  data | A pointer to a buffer where to store the element.
  stoptime | The absolute time at which to abort.

Returns
  Greater than or equal to zero on success, negative on failure.

## chirp_matrix_set()

int chirp_matrix_set | ( | struct chirp_matrix * | matrix,
|  | int | x,
|  | int | y,
|  | const void * | data,
|  | time_t | stoptime )

Set a single element.

Note: Writing a single element at a time is very inefficient. If possible, set multiple elements at once using chirp_matrix_set_row.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  x | The x position of the element.
  y | The y position of the element.
  data | A pointer to a buffer where to store the element.
  stoptime | The absolute time at which to abort.

Returns
  Greater than or equal to zero on success, negative on failure.

## chirp_matrix_setacl()

int chirp_matrix_setacl | ( | const char * | host,
|  | const char * | path,
|  | const char * | subject,
|  | const char * | rights,
|  | time_t | stoptime )

Set the acls on a matrix.

## chirp_matrix_width()

int chirp_matrix_width | ( | struct chirp_matrix * | matrix | ) |

Get the width of a matrix.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open

Returns
  The width of the matrix, measured in elements.

## chirp_matrix_height()

int chirp_matrix_height | ( | struct chirp_matrix * | matrix | ) |

Get the height of a matrix.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open

Returns
  The height of the matrix, measured in elements.

## chirp_matrix_element_size()

int chirp_matrix_element_size | ( | struct chirp_matrix * | matrix | ) |

Get the element size of a matrix.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open

Returns
  The size of each element in the matrix, measured in bytes.

## chirp_matrix_nhosts()

int chirp_matrix_nhosts | ( | struct chirp_matrix * | matrix | ) |

Get the number of hosts used by a matrix.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open

Returns
  The number of hosts used by the matrix.

## chirp_matrix_nfiles()

int chirp_matrix_nfiles | ( | struct chirp_matrix * | matrix | ) |

Get the number of files used by a matrix.

This value might be greater than the number of hosts, if the matrix is very large and it is necessary to break it into multiple files of one gigabyte or less.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open

Returns
  The number of files used by the matrix.

## chirp_matrix_fsync()

void chirp_matrix_fsync | ( | struct chirp_matrix * | matrix,
|  | time_t | stoptime )

Force all data to disk.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  stoptime | The absolute time at which to abort.

## chirp_matrix_close()

void chirp_matrix_close | ( | struct chirp_matrix * | matrix,
|  | time_t | stoptime )

Close a matrix and free all related resources.

Parameters

  matrix | A pointer to a chirp_matrix returned by chirp_matrix_create or chirp_matrix_open
  stoptime | The absolute time at which to abort.

## chirp_matrix_delete()

int chirp_matrix_delete | ( | const char * | host,
|  | const char * | path,
|  | time_t | stoptime )

Delete a matrix.

Parameters

  host | The hostname and optional port of the index file.
  path | The path to the index file.
  stoptime | The absolute time at which to abort.