ALLEN INSTITUTE

# bmtk.utils.sonata package

## Subpackages

- bmtk.utils.sonata.config package
  - Submodules
  - bmtk.utils.sonata.config.sonata_config module
    - `ConfigParser`
      - `ConfigParser.parse()`
      - `ConfigParser.usr_vars`
    - `SonataConfig`
      - `SonataConfig.from_dict()`
      - `SonataConfig.from_json()`
      - `SonataConfig.from_yaml()`
      - `SonataConfig.load()`
      - `SonataConfig.validate()`
      - `SonataConfig.validator`
    - `copy_config()`
    - `from_dict()`
    - `from_json()`
    - `from_yaml()`
  - Module contents

## Submodules

## bmtk.utils.sonata.column_property module

class bmtk.utils.sonata.column_property.ColumnProperty(name, dtype, dimension, nrows=0, attrs=None)

  Bases: `object`

  Representation of a column name and metadata from a hdf5 dataset, csv column, etc.

  property attributes
  property dimension
  property dtype
  classmethod from_csv(pd_obj, name=None)
  classmethod from_h5(hf_obj, name=None)
  property name
  property nrows

## bmtk.utils.sonata.edge module

class bmtk.utils.sonata.edge.Edge(src_node_id, trg_node_id, source_pop, target_pop, group_id, group_props, edge_types_props)

  Bases: `object`

  property dynamics_params
  property edge_type_id
  property group_id
  property source_node_id
  property source_population
  property target_node_id
  property target_population

class bmtk.utils.sonata.edge.EdgeSet(edge_ids, population)

  Bases: `object`

  next()

## bmtk.utils.sonata.edge_stats module

bmtk.utils.sonata.edge_stats.chisquare(edges_orig, edges_new, edge_prop, raw_counts=False, **kwargs)

bmtk.utils.sonata.edge_stats.edge_props_distribution(edge_files, edge_prop, populations=None, edge_props_grouping=None, source_props_grouping=None, target_props_grouping=None, fill_val=1, operation='sum', population_columns=False)

  Reads in one or more SONATA edges files and return a DataFrame consisting of the distribution of a given edge property across an arbitary grouping of cells. For example return the total number of synapses between each source/target node-type, or the mean syn_weights for edge edge-type, or the variance of connecting in-degrees across morphologies.

  By default will return a DataFrame of each “source_node_id”, “target_node_id”, and “<edge_prop>” row found in the edge file(s). But you can group the rows by any property/column found in the edges, source or target populations; and apply an arbitary <operation> on the results.

  The <edge_prop> can be any property/column found in the edges, target-nodes, or source-nodes populations. But if the property is not a numeric the <operation> applied to it may fail.

  Parameters:

    - edge_files – str, dict, list of str or dict. Is the edges (and optional nodes) files paths. It can be a SONATA h5 file, a dictionary of h5/csv files, or the location(s) of SONATA config files.

    - edge_prop – string, property name (column if h5 or csv) file that is being investigated.

    - populations – string or list of strings. If SONATA file(s) contains multiple edge populations you can specify .

    - edge_props_grouping – str or list[str]. List of columns in edges file(s) to group results by.

    - source_props_grouping – str or list[str]. List of columns in source-node file(s) to group results by.

    - target_props_grouping – str or list[str]. List of columns in target-node file(s) to group results by.

    - fill_val – If <edge_prop> has missing/None/NaN values will fill in with given value. set to None to Turn off.

    - operation – Str or function: pandas or numpy function to apply to when doing the grouping, eg. ‘sum’, ‘mean’, np.std.

    - population_columns – If set to true will return extra column describing the edges/nodes populations for each row.

bmtk.utils.sonata.edge_stats.edge_stats_table(edges_data)

bmtk.utils.sonata.edge_stats.kolmogorov_smirnov(edges_orig, edges_new, edge_prop, **kwargs)

bmtk.utils.sonata.edge_stats.nconnections_distributions(edge_files, populations=None, edge_props_grouping=None, source_props_grouping=None, target_props_grouping=None, **kwopts)

  Reads in one or more SONATA edges files and return a DataFrame consisting of the total number of connection given any arbitary grouping of network properties. The property will be called “nconns” in the returned table. Similar to edge_props_distribution().

  Note: A connection here just refers to whether-or-not two cells are connected (including autapses) and ignores the number of synapses between each cell. Use nsyns_distribution() to get distribution of the number of synapses.

  Parameters:

    - edge_files – str, dict, list of str or dict. Is the edges (and optional nodes) files paths. It can be a SONATA h5 file, a dictionary of h5/csv files, or the location(s) of SONATA config files.

    - populations – string or list of strings. If SONATA file(s) contains multiple edge populations you can specify .

    - source_props_grouping – str or list[str]. List of columns in source-node file(s) to group results by.

    - target_props_grouping – str or list[str]. List of columns in target-node file(s) to group results by.

bmtk.utils.sonata.edge_stats.nsyns_distribution(edge_files, populations=None, edge_props_grouping=None, source_props_grouping=None, target_props_grouping=None)

  Reads in one or more SONATA edges files and return a DataFrame consisting of the total number of synapses given any arbitary grouping of network properties. The property will be called “nsyns” in the returned table. Similar to edge_props_distribution().

  Note: Each cell-to-cell connection may have mutiliple synapses. Use nconnections_distributions() to get distribution of the raw connectivity map.

  Parameters:

    - edge_files – str, dict, list of str or dict. Is the edges (and optional nodes) files paths. It can be a SONATA h5 file, a dictionary of h5/csv files, or the location(s) of SONATA config files.

    - populations – string or list of strings. If SONATA file(s) contains multiple edge populations you can specify .

    - source_props_grouping – str or list[str]. List of columns in source-node file(s) to group results by.

    - target_props_grouping – str or list[str]. List of columns in target-node file(s) to group results by.

bmtk.utils.sonata.edge_stats.pearson_r(edges_orig, edges_new, edge_prop, **kwargs)

bmtk.utils.sonata.edge_stats.plot_correlation(edges_orig, edges_new, edge_prop, log_scale=False, ax=None, show=True)

bmtk.utils.sonata.edge_stats.plot_distribution(edges_data, edge_prop, names=None, log_scale=False, ax=None, show=True, **kwopts)

  Plots the distribution of an edge property given an arbitary grouping of rows based on edge, target or source node columns.

  Parameters:

    - edge_files – str, dict, list of str or dict. Is the edges (and optional nodes) files paths. It can be a SONATA h5 file, a dictionary of h5/csv files, or the location(s) of SONATA config files. Can also be a csv file containg results from edge_props_distribution() function

    - edge_prop – string, property name (column if h5 or csv) file that is being investigated.

    - names – A list of names/titles to use for the labeling of the distribution(s) curves. If None then will infer from data.

    - log_scale – If set to True then use log-scale on the x-axis. Default: False.

    - ax – If true will save distribution plot to pre-generated matplotlib axis, for merging into another figure. Default: None.

    - show – If true will plot distribution. Default: True.

    - kwargs – optional args that will be passed into edge_props_distribution() function.

bmtk.utils.sonata.edge_stats.to_edges_dataframe(edges_pop_h5, edge_types_path=None, with_properties=True)

bmtk.utils.sonata.edge_stats.to_nodes_dataframe(nodes_pop_h5, node_types_path=None, with_properties=True)

## bmtk.utils.sonata.file module

class bmtk.utils.sonata.file.File(data_files, data_type_files, mode='r', gid_table=None, require_magic=True)

  Bases: `object`

  property edges
  property has_edges
  property has_nodes
  property nodes
  property version

## bmtk.utils.sonata.file_root module

class bmtk.utils.sonata.file_root.EdgesRoot(edges, edge_types, mode='r')

  Bases: `FileRoot`

  property edge_types_table
  get_populations(name=None, source=None, target=None)

    Find all populations with matching criteria, either using the population name (which will return a list of size 0 or 1) or based on the source/target population.

    To return a list of all populations just use populations() method

    Parameters:

      - name – (str) name of population

      - source – (str or NodePopulation) returns edges with nodes coming from matching source-population

      - target – (str or NodePopulation) returns edges with nodes coming from matching target-population

    Returns:

      A (potential empty) list of EdgePopulation objects filter by criteria.

class bmtk.utils.sonata.file_root.FileRoot(root_name, h5_files, h5_mode, csv_files)

  Bases: `object`

  Base class for both /nodes and /edges root group in h5 file

  check_format()
  get_population(population_name, default=None)

    Return a population group object based on population’s name

  property population_names
  property populations
  property root_name
  property types_table

class bmtk.utils.sonata.file_root.NodesRoot(nodes, node_types, mode='r', gid_table=None)

  Bases: `FileRoot`

  generate_gids(file_name, gids=None, force=False)

    Creates a gid <–> (node_id, population) table based on sonnet specifications.

    > Generating gids will take some time and so not recommend to call this during the simulation. Instead save the file to the disk and pass in h5 file during the simulation (using gid_table parameter). In fact if you’re worried about efficeny don’t use this method.

    Parameters:

      - file_name – Name of h5 file to save gid map to.

      - gids – rule/list of gids to use

      - force – set to true to overwrite existing gid map (default False).

  property has_gids
  property node_types_table
  set_gid_table(gid_table, force=False)

    Adds a map from a gids <–> (node_id, population) based on specification.

    Parameters:

      - gid_table – An h5 file/group containing map specifications

      - force – Set to true to have it overwrite any exsiting gid table (default False)

## bmtk.utils.sonata.group module

class bmtk.utils.sonata.group.EdgeGroup(group_id, h5_group, parent)

  Bases: `Group`

  build_indicies(force=False)
  property edge_type_ids
  filter(**filter_props)
  get_values(property_name, all_rows=False)

    Returns all values for a group property.

    Note that a row within a group may not have a corresponding node/edge, or they may have a different order or multiple node/edges may share the same group row. Setting all_rows=False will return all the values as you see if you iterated through all the population’s items. Setting all_rows=True just returns the data as a list as they appear in the dataset (will be faster).

    Parameters:

      - property_name – Name of dataset property/column to fetch.

      - all_rows – Set false to return order in which they appear in population, false to return entire dataset

    Returns:

      A list of values for the given column name.

  next()
  property src_node_ids
  to_dataframe()
  property trg_node_ids

class bmtk.utils.sonata.group.Group(group_id, h5_group, parent)

  Bases: `object`

  A container containig a node/edge population groups.

  A node or edge population will have one or more groups, each having a unique identifier. Each group shared the same columns and datatypes, thus each group is essentially a different model.

  property all_columns
  build_indicies(force=False)
  check_format()
  column(column_name, group_only=False)
  property columns
  get_dataset(column_name)
  get_values(property_name, all_rows=False)

    Returns all values for a group property.

    Note that a row within a group may not have a corresponding node/edge, or they may have a different order or multiple node/edges may share the same group row. Setting all_rows=False will return all the values as you see if you iterated through all the population’s items. Setting all_rows=True just returns the data as a list as they appear in the dataset (will be faster).

    Parameters:

      - property_name – Name of dataset property/column to fetch.

      - all_rows – Set false to return order in which they appear in population, false to return entire dataset

    Returns:

      A list of values for the given column name.

  property group_columns
  property group_id
  property has_dynamics_params
  property has_gids
  property parent
  to_dataframe()

class bmtk.utils.sonata.group.NodeGroup(group_id, h5_group, parent)

  Bases: `Group`

  build_indicies(force=False)
  filter(**filter_props)

    Filter all nodes in the group by key=value pairs.

    The filter specifications may apply to either node_type or group column properties. Currently at the moment it only supports equivlency. An intersection (and operator) is done for every different filter pair. This will produce a generator of all nodes matching the the filters.

    for node in filter(pop_name=’VIp’, depth=10.0):

      assert(node[‘pop_name’] == ‘VIp’ and node[‘depth’] == 10.0)

    Parameters:

      filter_props – keys and their values to filter nodes on.

    Returns:

      A generator that produces all valid nodes within the group with matching key==value pairs.

  get_values(property_name, filtered_indicies=True)

    Returns all values for a group property.

    Note that a row within a group may not have a corresponding node/edge, or they may have a different order or multiple node/edges may share the same group row. Setting all_rows=False will return all the values as you see if you iterated through all the population’s items. Setting all_rows=True just returns the data as a list as they appear in the dataset (will be faster).

    Parameters:

      - property_name – Name of dataset property/column to fetch.

      - all_rows – Set false to return order in which they appear in population, false to return entire dataset

    Returns:

      A list of values for the given column name.

  property gids
  property node_ids
  property node_type_ids
  to_dataframe()

## bmtk.utils.sonata.node module

class bmtk.utils.sonata.node.Node(node_id, pop_name, node_type_id, node_types_props, group_id, group_props, dynamics_params, gid=None)

  Bases: `object`

  property dynamics_params
  property gid
  property group_id
  property group_props
  property node_id
  property node_type_id
  property node_type_properties
  property population_name

class bmtk.utils.sonata.node.NodeSet(node_indicies, population, **parameters)

  Bases: `object`

  get_properties(property_name)
  property gids
  next()
  property node_ids
  property node_type_ids

## bmtk.utils.sonata.population module

class bmtk.utils.sonata.population.EdgePopulation(pop_name, pop_group, edge_types_tables)

  Bases: `Population`

  build_indicies()
  property edge_types_table
  filter(**filter_props)
  get_row(index)
  get_source(source_node_id)
  static get_source_population(pop_group_h5)
  get_sources(source_node_ids)
  get_target(target_node_id)
  static get_target_population(pop_group_h5)
  get_targets(target_node_ids)
  property group_id_column
  property group_index_column
  group_indicies(group_id, build_cache=False, as_list=False)

    Returns a list of all the population row index that maps onto the given group.

    Used for iterating or searching within a Group

    Parameters:

      - group_id – id of a given group

      - build_cache – Will cache indicies for all groups. Will be faster if making multiple calls but requires more memory (default False)

    Returns:

      A (possibly empty) list of row indicies (non-contiguous, but unique)

  next()
  property source_population
  property target_population
  to_dataframe()

    Convert Population to dataframe

  property type_ids_column

class bmtk.utils.sonata.population.NodePopulation(pop_name, pop_group, node_types_tables)

  Bases: `Population`

  add_gids(gid_map_df, force=False)
  filter(**filter_props)
  get_gid(gid)
  get_node_id(node_id)
  get_row(row_indx)
  get_rows(row_indicies)

    Returns a set of all nodes based on list of row indicies.

    Warning: currently due to the use of h5py, the list must be ordered and cannot contain duplicates.

    Parameters:

      row_indicies – A list of row indicies

    Returns:

      An iterable NodeSet of nodes in the specified indicies

  property gids
  property group_id_column
  property group_index_column
  property has_gids
  igids(row_indicies)
  property index_column_name
  inode_ids(row_indicies)
  inode_type_ids(row_indicies)
  next()
  property node_ids
  property node_types_table
  to_dataframe(index_by_id=True)

    Convert Population to dataframe

  property type_ids_column

class bmtk.utils.sonata.population.Population(pop_name, pop_group, types_table)

  Bases: `object`

  get_group(group_id)
  property group_id_column
  property group_id_ds
  property group_ids

    List of all group_ids belonging to population

  property group_index_column
  property group_index_ds
  group_indicies(group_id, build_cache=False)

    Returns a list of all the population row index that maps onto the given group.

    Used for iterating or searching within a Group

    Parameters:

      - group_id – id of a given group

      - build_cache – Will cache indicies for all groups. Will be faster if making multiple calls but requires more memory (default False)

    Returns:

      A (possibly empty) list of row indicies (non-contiguous, but unique)

  property groups

    Returns a list of sonata.Group objects

  igroup_ids(row_indicies)
  igroup_indicies(row_indicies)
  property name

    name of current population

  to_dataframe()

    Convert Population to dataframe

  property type_ids
  property type_ids_column
  property types_table

## bmtk.utils.sonata.types_table module

class bmtk.utils.sonata.types_table.EdgeTypesTable(parent=None)

  Bases: `TypesTable`

  property edge_type_ids
  property index_column_name

class bmtk.utils.sonata.types_table.NodeTypesTable(parent=None)

  Bases: `TypesTable`

  property index_column_name
  property node_type_ids

class bmtk.utils.sonata.types_table.TypesTable(parent=None)

  Bases: `object`

  add_table(nt_df)
  column(column_name)
  property columns
  find(column_key, column_val, silent=False)

    Returns a list of type_ids that contain column property column_key==column_val

    Parameters:

      - column_key – Name of column to search

      - column_val – Value of column to select for

      - silent – Set to true to prevent KeyError if column_key doesn’t exist (default=False)

    Returns:

      A (potentially empty) list of type_ids

  property index_column_name
  next()
  to_dataframe(cache=False)
  property type_ids

bmtk.utils.sonata.types_table.remove_nans(types_dict)

  Convert nan values to None in type row (dict)

## bmtk.utils.sonata.utils module

bmtk.utils.sonata.utils.add_hdf5_magic(hdf5_handle)

bmtk.utils.sonata.utils.add_hdf5_version(hdf5_handle)

bmtk.utils.sonata.utils.check_magic(hdf5_file)

  Check the magic attribute exists according to the sonata format

bmtk.utils.sonata.utils.get_attribute_h5(h5obj, attribut_name, default=None)

bmtk.utils.sonata.utils.get_node_ids(nodes_path, population)

bmtk.utils.sonata.utils.get_version(hdf5_file)

bmtk.utils.sonata.utils.listify(files)

bmtk.utils.sonata.utils.load_csv(csvfile)

bmtk.utils.sonata.utils.load_h5(h5file, mode='r')

## Module contents