# API and Code

Below is the API documention for the the Stetl Python code.

## Main Entry Points

There are several entry points through which Stetl can be called. The most common is to use the commandline script bin/stetl. This command should be available after doing an install.

In some contexts like integrations you may want to call Stetl via Python. The entries are then.

`stetl.main.``main`()

  The main function, to be called from commandline, like python src/main.py -c etl.cfg.

  Args:
    `-c`
      `--config <config_file>`

        the Stetl config file.

    `-s`

      –section <section_name> the section in the Stetl config (ini) file to execute (default is [etl]).

    `-a`

      –args <arglist> sero or more substitutable args for symbolic, {arg}, values in Stetl config file, in format -a arg1=foo -a arg2=bar etc.

    `-d`

      –doc <class> Get component documentation like its configuration parameters, e.g. stetl –doc stetl.inputs.fileinput.FileInput

    `-v`

      –version Show the current version of stelt and exit

    `-h`

      –help get help info

`stetl.main.``print_config_attrs`(clazz)

  Deprecated since version 1.0.7.

  now using @Config which also documents with Spinx.

`stetl.main.``print_doc`(class_name)

  Print documentation for class in particular config options

  Deprecated since version 1.0.7: now using @Config which also documents with Spinx.

class `stetl.etl.``ETL`(options_dict, args_dict=None)

  The main class: builds ETL Chains with connected Components from a config and let them run.

  Usually this class is called via `main` but it may be called directly for direct integration.

  `env_expand_args_dict`(args_dict, args_names)

    Expand values in dict with equivalent values from the OS Env. NB vars in OS Env should be prefixed with STETL_ or stetl_ as to get overrides by accident.

    Returns

      expanded args_dict or None

## Core Framework

The core framework is directly under the directory src/stetl. Below are the main seven classes. Their interrelation is as follows:

One or more `stetl.chain.Chain` objects are built from a Stetl ETL configuration via the `stetl.factory.Factory` class. A `stetl.chain.Chain` consists of a set of connected `stetl.component.Component` objects. A `stetl.component.Component` is either an `stetl.input.Input`, an `stetl.output.Output` or a `stetl.filter.Filter`. Data and status flows as `stetl.packet.Packet` objects from an `stetl.input.Input` via zero or more `stetl.filter.Filter` objects to a final `stetl.output.Output`.

As a trivial example: an `stetl.input.Input` could be an XML file, a `stetl.filter.Filter` could represent an XSLT file and an `stetl.output.Output` a PostGIS database. This is effected by specialized classes in the subpackages inputs, filters, and outputs. New in 1.1.0: `stetl.Splitter` to split data to multiple Outputs and `stetl.Merger` to combine multiple Inputs.

class `stetl.factory.``Factory`

  Object and class Factory (Pattern). Based on: http://stackoverflow.com/questions/2226330/instantiate-a-python-class-from-a-name

  `class_forname`(class_string)

    Returns class instance specified by a string.

    Args:

      class_string: The string representing a class.

    Raises:

      ValueError if module part of the class is not specified.

  `new_instance`(class_obj, configdict, section)

    Returns object instance from class instance.

    Args:

      class_obj: object representing a class instance. args: standard args. kwargs: standard args.

class `stetl.component.``Component`(configdict, section, consumes='none', produces='none')

  Abstract Base class for all Input, Filter and Output Components.

  `after_chain_invoke`(packet)

    Called right after entire Component Chain invoke.

  `after_invoke`(packet)

    Called right after Component invoke.

  `before_invoke`(packet)

    Called just before Component invoke.

  `exit`()

    Allows derived Components to perform a one-time exit/cleanup.

  `init`()

    Allows derived Components to perform a one-time init.

  `input_format`()
    `CONFIG`

      The specific input format if the consumes parameter is a list or the format to be converted to the output_format.

    - type: str

    - required: False

    - default: None

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

  `output_format`()
    `CONFIG`

      The specific output format if the produces parameter is a list or the format to which the input format is converted.

    - type: str

    - required: False

    - default: None

  `timer_stop`(start_time)

    Collect and calculate per-Component performance timing stats. :param start_time: :return:

class `stetl.component.``Config`(ptype=<class 'str'>, default=None, required=False)

  Decorator class to tie config values from the .ini file to object instance property values. Somewhat like the Python standard @property but with the possibility to define default values, typing and making properties required.

  Each property is defined by @Config(type, default, required). Basic idea comes from: https://wiki.python.org/moin/PythonDecoratorLibrary#Cached_Properties

class `stetl.chain.``Chain`(chain_str, config_dict)

  Holder for single invokable pipeline of components A Chain is basically a singly linked list of Components Each Component executes a part of the total ETL. Data along the Chain is passed within a Packet object. The compatibility of input and output for linked Components is checked when adding a Component to the Chain.

  `add`(etl_comp)

    Add component to end of Chain :param etl_comp: :return:

  `assemble`()

    Builder method: build a Chain of linked Components :return:

  `get_by_class`(clazz)

    Get Component instance from Chain by class, mainly for testing. :param clazz: :return Component:

  `get_by_id`(id)

    Get Component instance from Chain, mainly for testing. :param name: :return Component:

  `get_by_index`(index)

    Get Component instance from Chain by position/index in Chain, mainly for testing. :param clazz: :return Component:

  `run`()

    Run the ETL Chain. :return:

class `stetl.packet.``FORMAT`

  Format of Packet (enumeration).

  Current possible values:

  - ‘none’

  - ‘xml_line_stream’

  - ‘line_stream’

  - ‘etree_doc’

  - ‘etree_element’

  - ‘etree_feature_array’

  - ‘xml_doc_as_string’

  - ‘string’

  - ‘record’

  - ‘record_array’

  - ‘struct’

  - ‘geojson_feature’

  - ‘geojson_collection’

  - ‘ogr_feature’

  - ‘ogr_feature_array’

  - ‘any’

class `stetl.packet.``Packet`(data=None)

  Represents units of (any) data and status passed along Chain of Components.

class `stetl.input.``Input`(configdict, section, produces)

  Bases: `stetl.component.Component`

  Abstract Base class for all Input Components.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.output.``Output`(configdict, section, consumes)

  Bases: `stetl.component.Component`

  Abstract Base class for all Output Components.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filter.``Filter`(configdict, section, consumes, produces)

  Bases: `stetl.component.Component`

  Maps input to output. Abstract base class for specific Filters.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.splitter.``Splitter`(config_dict, child_list)

  Bases: `stetl.component.Component`

  Component that splits a single input to multiple output Components. Use this for example to produce multiple output file formats (GML, GeoJSON etc) or to publish to multiple remote services (SOS, SensorThings API) or for simple debugging: target Output and StandardOutput.

  `after_chain_invoke`(packet)

    Called right after entire Component Chain invoke.

  `after_invoke`(packet)

    Called right after Component invoke.

  `before_invoke`(packet)

    Called just before Component invoke.

  `exit`()

    Allows derived Components to perform a one-time exit/cleanup.

  `init`()

    Allows derived Components to perform a one-time init.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.merger.``Merger`(config_dict, child_list)

  Bases: `stetl.component.Component`

  Component that merges multiple Input Components into a single Component. Use this for example to combine multiple input streams like API endpoints. The Merger will embed Child Components to which actions are delegated. A Child Component may be a sub-Chain e.g. (Input|Filter|Filter..) sequence. Hence the “next” should be coupled to the last Component in that sub-Chain with the degenerate case where the sub-Chain is a single (Input) Component. NB this Component can only be used for Inputs.

  `first`(child)

    Get first Component in Child sub-Chain. :param child: :return: first Component

  `last`(child)

    Get last Component in Child sub-Chain. :param child: :return: last Component

## Components: Inputs

class `stetl.inputs.dbinput.``DbInput`(configdict, section, produces)

  Bases: `stetl.input.Input`

  Input from any database (abstract base class).

class `stetl.inputs.dbinput.``PostgresDbInput`(configdict, section)

  Bases: `stetl.inputs.dbinput.SqlDbInput`

  Input by querying records from a Postgres database. Input is a query, like SELECT * from mytable. Output is zero or more records as record array (array of dict) or single record (dict).

  produces=FORMAT.record_array (default) or FORMAT.record

  `exit`()

    Allows derived Components to perform a one-time exit/cleanup.

  `host`()
    `CONFIG`

      host name or host IP-address, defaults to ‘localhost’

    - type: str

    - required: False

    - default: localhost

  `init`()

    Allows derived Components to perform a one-time init.

  `password`()
    `CONFIG`

      User password, defaults to ‘postgres’

    - type: str

    - required: False

    - default: postgres

  `port`()
    `CONFIG`

      port for host, defaults to ‘5432’

    - type: str

    - required: False

    - default: 5432

  `raw_query`(query_str)

    Performs DB-specific query and returns raw records iterator.

  `schema`()
    `CONFIG`

      The postgres schema name, defaults to ‘public’

    - type: str

    - required: False

    - default: public

  `user`()
    `CONFIG`

      User name, defaults to ‘postgres’

    - type: str

    - required: False

    - default: postgres

class `stetl.inputs.dbinput.``SqlDbInput`(configdict, section)

  Bases: `stetl.inputs.dbinput.DbInput`

  Input using a query from any SQL-based RDBMS (abstract base class).

  `column_names`()
    `CONFIG`

      Column names to populate records with. If empty taken from table metadata.

    - type: str

    - required: False

    - default: None

  `database_name`()
    `CONFIG`

      Database name

    - type: str

    - required: True

    - default: None

  `do_query`(query_str)

    DB-neutral query returning Python record list.

  `query`()
    `CONFIG`

      The query (string) to fire.

    - type: str

    - required: False

    - default: None

  `raw_query`(query_str)

    Performs DB-specific query and returns raw records iterator.

  `read_once`()
    `CONFIG`

      Read once? i.e. only do query once and stop

    - type: bool

    - required: False

    - default: False

  `result_to_output`(db_tuples)

    Convert DB-specific record tuples to single Python record (dict) or record array (list of dict).

  `table`()
    `CONFIG`

      Table name

    - type: str

    - required: False

    - default: None

  `tuples_to_records`(db_tuples, columns=None)

    Convert tuple array (list of tuple) to list of records (list of dict’s) using list of column names.

class `stetl.inputs.dbinput.``SqliteDbInput`(configdict, section)

  Bases: `stetl.inputs.dbinput.SqlDbInput`

  Input by querying records from a SQLite database. Input is a query, like SELECT * from mytable. Output is zero or more records as record array (array of dict) or single record (dict).

  produces=FORMAT.record_array (default) or FORMAT.record

  `init`()

    Allows derived Components to perform a one-time init.

  `raw_query`(query_str)

    Performs DB-specific query and returns raw records iterator.

class `stetl.inputs.fileinput.``ApacheLogFileInput`(configdict, section)

  Bases: `stetl.inputs.fileinput.FileInput`

  Parses Apache log files. Lines are converted into records based on the log format. Log format should follow Apache Log Format. See ApacheLogParser for details.

  produces=FORMAT.record

  `key_map`()
    `CONFIG`

      Map of cryptic %-field names to readable keys in record.

    - type: dict

    - required: False

    - default: {‘%l’: ‘logname’, ‘%>s’: ‘status’, ‘%D’: ‘deltat’, ‘%{User-agent}i’: ‘agent’, ‘%b’: ‘bytes’, ‘%{Referer}i’: ‘referer’, ‘%u’: ‘user’, ‘%t’: ‘time’, “’%h”: ‘host’, ‘%r’: ‘request’}

  `log_format`()
    `CONFIG`

      Log format according to Apache CLF

    - type: str

    - required: False

    - default: %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"

class `stetl.inputs.fileinput.``CsvFileInput`(configdict, section)

  Bases: `stetl.inputs.fileinput.FileInput`

  Parse CSV file into stream of records (dict structures) or a one-time record array. NB raw version: CSV needs to have first line with fieldnames.

  produces=FORMAT.record or FORMAT.record_array

  `delimiter`()
    `CONFIG`

      A one-character string used to separate fields. It defaults to ‘,’.

    - type: str

    - required: False

    - default: ,

  `init`()

    Allows derived Components to perform a one-time init.

  `quote_char`()
    `CONFIG`

      A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters. It defaults to ‘”’.

    - type: str

    - required: False

    - default: “

class `stetl.inputs.fileinput.``FileInput`(configdict, section, produces)

  Bases: `stetl.input.Input`

  Abstract base class for specific FileInputs, use derived classes.

  `depth_search`()
    `CONFIG`

      Should we recurse into sub-directories to find files?

    - type: bool

    - required: False

    - default: False

  `file_path`()
    `CONFIG`

      Path to file or files or URLs: can be a dir or files or URLs or even multiple, comma separated. For URLs only JSON is supported now.

    - type: str

    - required: False

    - default: None

  `filename_pattern`()
    `CONFIG`

      Filename pattern according to Python `glob.glob` for example: ‘*.[gxGX][mM][lL]’

    - type: str

    - required: False

    - default: *.[gxGX][mM][lL]

  `read_file`(file_path)

    Override in subclass.

class `stetl.inputs.fileinput.``GlobFileInput`(configdict, section, produces=['string', 'line_stream'])

  Bases: `stetl.inputs.fileinput.FileInput`

  Returns file names based on the glob.glob pattern given as filename_filter.

  produces=FORMAT.string or FORMAT.line_stream

class `stetl.inputs.fileinput.``JsonFileInput`(configdict, section)

  Bases: `stetl.inputs.fileinput.FileInput`

  Parse JSON file from file system or URL into hierarchical data struct. The struct format may also be a GeoJSON structure. In that case the output_format needs to be explicitly set to geojson_collection in the component config.

  produces=FORMAT.struct or FORMAT.geojson_collection

  `read_file`(file_path)

    Override in subclass.

class `stetl.inputs.fileinput.``LineStreamerFileInput`(configdict, section, produces='line_stream')

  Bases: `stetl.inputs.fileinput.FileInput`

  Reads text-files, producing a stream of lines, one line per Packet. NB assumed is that lines in the file have newlines !!

  `process_line`(line)

    Override in subclass.

class `stetl.inputs.fileinput.``StringFileInput`(configdict, section)

  Bases: `stetl.inputs.fileinput.FileInput`

  Reads and produces file as String.

  produces=FORMAT.string

  `format_args`()
    `CONFIG`

      Formatting of content according to Python String.format() Input file should have substitutable values like {schema} {foo} format_args should be of the form `format_args = schema:test foo:bar`

    - type: str

    - required: False

    - default: None

  `read_file`(file_path)

    Overridden from base class.

class `stetl.inputs.fileinput.``XmlElementStreamerFileInput`(configdict, section)

  Bases: `stetl.inputs.fileinput.FileInput`

  Extracts XML elements from a file, outputs each feature element in Packet. Parsing is streaming (no internal DOM buildup) so any file size can be handled. Use this class for your big GML files!

  produces=FORMAT.etree_element

  `element_tags`()
    `CONFIG`

      Comma-separated string of XML (feature) element tag names of the elements that should be extracted and added to the output element stream.

    - type: list

    - required: True

    - default: None

  `strip_namespaces`()
    `CONFIG`

      should namespaces be removed from the input document and thus not be present in the output element stream?

    - type: bool

    - required: False

    - default: False

class `stetl.inputs.fileinput.``XmlFileInput`(configdict, section)

  Bases: `stetl.inputs.fileinput.FileInput`

  Parses XML files into etree docs (do not use for large files!).

  produces=FORMAT.etree_doc

  `read_file`(file_path)

    Override in subclass.

class `stetl.inputs.fileinput.``XmlLineStreamerFileInput`(*args, **kwargs)

  Bases: `stetl.inputs.fileinput.LineStreamerFileInput`

  DEPRECATED Streams lines from an XML file(s) NB assumed is that lines in the file have newlines !!

  produces=FORMAT.xml_line_stream

  Deprecated since version 1.0.9: Better to use XmlElementStreamerFileInput for GML features

class `stetl.inputs.fileinput.``ZipFileInput`(configdict, section)

  Bases: `stetl.inputs.fileinput.FileInput`

  Parse ZIP file from file system or URL into a stream of records containing zipfile-path and file names.

  produces=FORMAT.record

  `name_filter`()
    `CONFIG`

      Regular “glob.glob” expression for filtering out filenames from the ZIP archive.

    - type: str

    - required: False

    - default: *

class `stetl.inputs.httpinput.``ApacheDirInput`(configdict, section, produces='record')

  Bases: `stetl.inputs.httpinput.HttpInput`

  Read file data from an Apache directory “index” HTML page. Uses http://stackoverflow.com/questions/686147/url-tree-walker-in-python produces=FORMAT.record. Each record contains file_name and file_data (other meta data like date time is too fragile over different Apache servers).

  `file_ext`()
    `CONFIG`

      The file extension for target files in Apache dir.

    - type: str

    - required: False

    - default: xml

  `filter_file`(file_name)

    Filter the file_name, e.g. to suppress reading, default: return file_name.

    Parameters

      file_name –

    Return string or None

  `init`()

    Read the list of files from the Apache index URL.

  `next_file`()

    Return a tuple (name, date, size) with next file info.

    Return tuple

  `no_more_files`()

    More files left?.

    Return Boolean

  `read`(packet)

    Read the data from the URL.

    Parameters

      packet –

    Returns

class `stetl.inputs.httpinput.``HttpInput`(configdict, section, produces='any')

  Bases: `stetl.input.Input`

  Fetch data from remote services like WFS via HTTP protocol. Base class: subclasses will do datatype-specific formatting of the returned data.

  produces=FORMAT.any

  `add_authorization`(request)

    Add authorization from config data. Authorization scheme-specific. May be extended or overloaded for additional schemes.

    Parameters

      request – the HTTP Request

    Returns

  `auth`()
    `CONFIG`

      Authentication data: Flat JSON-like struct dependent on auth type/schema. Only the type field is required, other fields depend on auth schema. Supported values :

          type: basic|token

      If the type is `basic` (HTTP Basic Authentication) two additional fields `user` and `password` are required. If the type is `token` (HTTP Token) additional two additional fields `keyword` and `token` are required.

      Any required Base64 encoding is provided by `HttpInput`.

      Examples:

          # Basic Auth
          url = https://some.rest.api.com
          auth = {
              type: basic,
              user: myname
              password: mypassword
          }

          # Token Auth
          url = https://some.rest.api.com
          auth = {
              type: token,
              keyword: Bearer
              token: mytoken
          }

    - type: dict

    - required: False

    - default: None

  `format_data`(data)

    Format response data, override in subclasses, defaults to returning original data. :param packet: :return:

  `parameters`()
    `CONFIG`

      Flat JSON-like struct of the parameters to be appended to the url.

      Example: (parameters require quotes):

          url = http://geodata.nationaalgeoregister.nl/natura2000/wfs
          parameters = {
              service : WFS,
              version : 1.1.0,
              request : GetFeature,
              srsName : EPSG:28992,
              outputFormat : text/xml; subtype=gml/2.1.2,
              typename : natura2000
          }

    - type: dict

    - required: False

    - default: None

  `read`(packet)

    Read the data from the URL.

    Parameters

      packet –

    Returns

  `read_from_url`(url, parameters=None)

    Read the data from the URL.

    Parameters

      - url – the url to fetch

      - parameters – optional dict of query parameters

    Returns

  `url`()
    `CONFIG`

      The HTTP URL string.

    - type: str

    - required: True

    - default: None

class `stetl.inputs.ogrinput.``OgrInput`(configdict, section)

  Bases: `stetl.input.Input`

  Direct GDAL OGR input via Python OGR wrapper. Via the Python API http://gdal.org/python an OGR data source is accessed and from each layer the Features are read. Each Layer corresponds to a “doc”, so for multi-layer sources the ‘end-of-doc’ flag is set after a Layer has been read.

  This input can read almost any geospatial dataformat. One can use the features directly in a Stetl Filter or use a converter to e.g. convert to GeoJSON structures.

  produces=FORMAT.ogr_feature or FORMAT.ogr_feature_array (all features)

  `data_source`()
    `CONFIG`

      String denoting the OGR datasource. Usually a path to a file like “path/rivers.shp” or connection string to PostgreSQL like “PG: host=localhost dbname=’rivers’ user=’postgres’”.

    - type: str

    - required: True

    - default: None

  `init`()

    Allows derived Components to perform a one-time init.

  `source_format`()
    `CONFIG`

      Instructs GDAL to use driver by that name to open datasource. Not required for many standard formats that are self-describing like ESRI Shapefile.

      Examples: ‘PostgreSQL’, ‘GeoJSON’ etc

    - type: str

    - required: False

    - default: None

  `source_options`()
    `CONFIG`

      Custom datasource-specific options. Used in gdal.SetConfigOption().

    - type: dict

    - required: False

    - default: None

  `sql`()
    `CONFIG`

      String with SQL query. Mandatory for PostgreSQL OGR source.

    - type: str

    - required: False

    - default: None

class `stetl.inputs.ogrinput.``OgrPostgisInput`(configdict, section)

  Bases: `stetl.input.Input`

  Input from PostGIS via ogr2ogr command. For now hardcoded to produce an ogr GML line stream. OgrInput may be a better alternative.

  Alternatives: either stetl.input.PostgresqlInput or stetl.input.OgrInput.

  produces=FORMAT.xml_line_stream

  `in_pg_db`()
    `CONFIG`

      Database name input DB.

    - type: str

    - required: True

    - default: None

  `in_pg_host`()
    `CONFIG`

      Host of input DB.

    - type: str

    - required: False

    - default: localhost

  `in_pg_password`()
    `CONFIG`

      Password input DB.

    - type: str

    - required: False

    - default: postgres

  `in_pg_port`()
    `CONFIG`

      Port of input DB.

    - type: str

    - required: False

    - default: 5432

  `in_pg_schema`()
    `CONFIG`

      DB Schema name input DB.

    - type: str

    - required: False

    - default: None

  `in_pg_sql`()
    `CONFIG`

      The input query (string) to fire.

    - type: str

    - required: False

    - default: None

  `in_pg_user`()
    `CONFIG`

      User input DB.

    - type: str

    - required: False

    - default: postgres

  `in_srs`()
    `CONFIG`

      SRS (projection) (ogr2ogr -s_srs) input DB e.g. ‘EPSG:28992’.

    - type: str

    - required: False

    - default: None

  `init`()

    Allows derived Components to perform a one-time init.

  `out_dimension`()
    `CONFIG`

      Dimension (OGR: DIM=N) of features in output stream.

    - type: str

    - required: False

    - default: 2

  `out_geotype`()
    `CONFIG`

      OGR Geometry type new layer in output stream, e.g. POINT.

    - type: str

    - required: False

    - default: None

  `out_gml_format`()
    `CONFIG`

      GML format OGR name in output stream, e.g. ‘GML3’.

    - type: str

    - required: False

    - default: None

  `out_layer_name`()
    `CONFIG`

      New Layer name (ogr2ogr -nln) output stream, e.g. ‘address’.

    - type: str

    - required: False

    - default: None

  `out_srs`()
    `CONFIG`

      Target SRS (ogr2ogr -t_srs) code output stream.

    - type: str

    - required: False

    - default: None

## Components: Filters

class `stetl.filters.xsltfilter.``XsltFilter`(configdict, section)

  Bases: `stetl.filter.Filter`

  Invokes XSLT processor (via lxml) for given XSLT script on an etree doc.

  consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

  `script`()
    `CONFIG`

      Path to XSLT script file.

    - type: str

    - required: True

    - default: None

class `stetl.filters.xmlassembler.``XmlAssembler`(configdict, section)

  Bases: `stetl.filter.Filter`

  Split a stream of etree DOM XML elements (usually Features) into etree DOM docs. Consumes and buffers elements until max_elements reached, will then produce an etree doc.

  consumes=FORMAT.etree_element, produces=FORMAT.etree_doc

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.xmlelementreader.``XmlElementReader`(configdict, section)

  Bases: `stetl.filter.Filter`

  Extracts XML elements from a file, outputs each feature element in Packet. Parsing is streaming (no internal DOM buildup) so any file size can be handled. Use this class for your big GML files!

  consumes=FORMAT.string, produces=FORMAT.etree_element

  `element_tags`()
    `CONFIG`

      Comma-separated string of XML (feature) element tag names of the elements that should be extracted and added to the output element stream.

    - type: list

    - required: True

    - default: None

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

  `strip_namespaces`()
    `CONFIG`

      should namespaces be removed from the input document and thus not be present in the output element stream?

    - type: bool

    - required: False

    - default: False

class `stetl.filters.xmlvalidator.``XmlSchemaValidator`(configdict, section)

  Bases: `stetl.filter.Filter`

  Validates an etree doc and prints result to log.

  consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.sieve.``AttrValueRecordSieve`(configdict, section)

  Bases: `stetl.filters.sieve.Sieve`

  Sieves by attr/value(s) in Record Packets.

  `attr_name`()
    `CONFIG`

      Name of attribute whose value(s) are to be sieved.

    - type: str

    - required: True

    - default: None

  `attr_values`()
    `CONFIG`

      Value(s) for attribute to be to sieved. If empty any value is passed through (existence of attr_name is criterium).

    - type: list

    - required: False

    - default: []

  `sieve`(packet)

    Filter out Packets that are not matching designated attr value(s). :param packet: :return:

class `stetl.filters.sieve.``Sieve`(configdict, section, consumes, produces)

  Bases: `stetl.filter.Filter`

  ABC for specific Sieves that pass-through, “sieve”, Packets based on criteria in their data.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

  `sieve`(packet)

    To be implemented in subclasses. :param packet: :return:

class `stetl.filters.stringfilter.``StringFilter`(configdict, section, consumes, produces)

  Bases: `stetl.filter.Filter`

  Base class for any string filtering

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.stringfilter.``StringSubstitutionFilter`(configdict, section)

  Bases: `stetl.filters.stringfilter.StringFilter`

  String filtering using Python advanced String formatting. String should have substitutable values like {schema} {foo} format_args should be of the form format_args = schema:test foo:bar …

  consumes=FORMAT.string, produces=FORMAT.string

  `format_args`()
    `CONFIG`

      Provides a list of format arguments used by the string substitution filter. Formatting of content according to Python String.format(). String should have substitutable values like {schema} {foo}.

      Example: format_args = schema:test foo:bar

    - type: str

    - required: True

    - default: None

  `separator`()
    `CONFIG`

      Provides the separator to split the format argument names from their values.

    - type: str

    - required: False

    - default: :

class `stetl.filters.templatingfilter.``Jinja2TemplatingFilter`(configdict, section)

  Bases: `stetl.filters.templatingfilter.TemplatingFilter`

  Implements Templating using Jinja2. Jinja2 http://jinja.pocoo.org, is a modern and designer-friendly templating language for Python modelled after Django’s templates. A ‘struct’ format as input provides a tree-like structure that could originate from a JSON file or REST service. This input struct provides all the variables to be inserted into the template. The template itself can be configured in this component as a Jinja2 string or -file. An optional ‘template_search_paths’ provides a list of directories from which templates can be fethced. Default is the current working directory. Via the optional ‘globals_path’ a JSON structure can be inserted into the Template environment. The variables in this globals struture are typically “boilerplate” constants like: id-prefixes, point of contacts etc.

  consumes=FORMAT.struct, produces=FORMAT.string

  `add_env_filters`(jinja2_env)

    Register additional Filters on the template environment by updating the filters dict: Somehow min and max of list are not present so add them as well.

  `create_template`()

    To be overridden in subclasses.

  static `geojson2gml_filter`(value, source_crs=4326, target_crs=None, gml_id=None, gml_format='GML2', gml_longsrs='NO')

    Jinja2 custom Filter: generates any GML geometry from a GeoJSON geometry. By specifying a target_crs we can even reproject from the source CRS. The gml_format=GML2|GML3 determines the general GML form: e.g. pos/posList or coordinates. gml_longsrs=YES|NO determines the srsName format like EPSG:4326 or urn:ogc:def:crs:EPSG::4326 (long).

  `template_globals_path`()
    `CONFIG`

      One or more JSON files or URLs with global variables that can be used anywhere in template. Multiple files will be merged into one globals dictionary

    - type: str

    - required: False

    - default: None

  `template_search_paths`()
    `CONFIG`

      List of directories where to search for templates, default is current working directory only.

    - type: str

    - required: False

    - default: None

class `stetl.filters.templatingfilter.``StringTemplatingFilter`(configdict, section)

  Bases: `stetl.filters.templatingfilter.TemplatingFilter`

  Implements Templating using Python’s internal string.Template. A template string or file should be configured. The input record contains the actual values to be substituted in the template string as a record (key/value pairs). Output is a regular string.

  consumes=FORMAT.record or FORMAT.record_array, produces=FORMAT.string

  `create_template`()

    To be overridden in subclasses.

  `safe_substitution`()
    `CONFIG`

      Apply safe substitution? With this method, string.Template.safe_substitute will be invoked, instead of string.Template.substitute. If placeholders are missing from mapping and keywords, instead of raising an exception, the original placeholder will appear in the resulting string intact.

    - type: bool

    - required: False

    - default: False

class `stetl.filters.templatingfilter.``TemplatingFilter`(configdict, section, consumes='any', produces='string')

  Bases: `stetl.filter.Filter`

  Abstract base class for specific template-based filters. See https://wiki.python.org/moin/Templating Subclasses implement a specific template language like Python string.Template, Mako, Genshi, Jinja2,

  consumes=FORMAT.any, produces=FORMAT.string

  `create_template`()

    To be overridden in subclasses.

  `exit`()

    Allows derived Components to perform a one-time exit/cleanup.

  `init`()

    Allows derived Components to perform a one-time init.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

  `template_file`()
    `CONFIG`

      Path to template file. One of template_file or template_string needs to be configured.

    - type: str

    - required: False

    - default: None

  `template_string`()
    `CONFIG`

      Template string. One of template_file or template_string needs to be configured.

    - type: str

    - required: False

    - default: None

class `stetl.filters.gmlfeatureextractor.``GmlFeatureExtractor`(configdict, section='gml_feature_extractor')

  Bases: `stetl.filter.Filter`

  Extract arrays of GML features etree elements from etree docs.

  consumes=FORMAT.etree_doc, produces=FORMAT.etree_feature_array

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.formatconverter.``FormatConverter`(configdict, section)

  Bases: `stetl.filter.Filter`

  Converts (almost) any packet format (if converter available).

  consumes=FORMAT.any, produces=FORMAT.any but actual formats are changed at initialization based on the input to output format to be converted via the input_format and output_format config parameters.

  `converter_args`()
    `CONFIG`

      Custom converter-specific arguments.

    - type: dict

    - required: False

    - default: None

  static `etree_doc2geojson_collection`(packet, converter_args=None)

    Use converter_args to determine XML tag names for features and GeoJSON feature id. For example

    > converter_args = {

    >   ‘root_tag’: ‘FeatureCollection’, ‘feature_tag’: ‘featureMember’, ‘feature_id_attr’: ‘fid’ }

    Parameters

      - packet –

      - converter_args –

    Returns

  static `etree_doc2struct`(packet, strip_space=True, strip_ns=True, sub=False, attr_prefix='', gml2ogr=True, ogr2json=True)
    Parameters

      - packet –

      - strip_space –

      - strip_ns –

      - sub –

      - attr_prefix –

      - gml2ogr –

      - ogr2json –

    Returns

  static `etree_elem2geojson_feature`(packet, converter_args=None)
  static `etree_elem2struct`(packet, strip_space=True, strip_ns=True, sub=False, attr_prefix='', gml2ogr=True, ogr2json=True)
    Parameters

      - packet –

      - strip_space –

      - strip_ns –

      - sub –

      - attr_prefix –

      - gml2ogr –

      - ogr2json –

    Returns

  `init`()

    Allows derived Components to perform a one-time init.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.execfilter.``CommandExecFilter`(configdict, section)

  Bases: `stetl.filters.execfilter.ExecFilter`

  Executes an arbitrary command and captures the output

  consumes=FORMAT.string, produces=FORMAT.string

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.execfilter.``ExecFilter`(configdict, section, consumes, produces)

  Bases: `stetl.filter.Filter`

  Executes any command (abstract base class).

  `env_args`()
    `CONFIG`

      Provides of list of environment variables which will be used when executing the given command.

      Example: env_args = pgpassword=postgres othersetting=value~with~spaces

    - type: str

    - required: False

    - default:

  `env_separator`()
    `CONFIG`

      Provides the separator to split the environment variable names from their values.

    - type: str

    - required: False

    - default: =

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.nullfilter.``NullFilter`(configdict, section, consumes='any', produces='any')

  Bases: `stetl.filter.Filter`

  Pass-through Filter, does nothing. Mainly used in Test Cases.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.packetbuffer.``PacketBuffer`(configdict, section)

  Bases: `stetl.filter.Filter`

  Buffers all incoming Packets, main use is unit-testing to inspect Packets after ETL is done.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.packetwriter.``PacketWriter`(configdict, section)

  Bases: `stetl.filter.Filter`

  Writes the payload of a packet as a string to a file.

  consumes=FORMAT.any, produces=FORMAT.string

  `file_path`()
    `CONFIG`

      File path to write content to.

    - type: str

    - required: True

    - default: None

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

class `stetl.filters.regexfilter.``RegexFilter`(configdict, section, consumes='string', produces='record')

  Bases: `stetl.filter.Filter`

  Extracts data from a string using a regular expression and returns the named groups as a record. consumes=FORMAT.string, produces=FORMAT.record

  `exit`()

    Allows derived Components to perform a one-time exit/cleanup.

  `init`()

    Allows derived Components to perform a one-time init.

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

  `pattern_string`()
    `CONFIG`

      Regex pattern string. Should contain named groups.

    - type: str

    - required: True

    - default: None

class `stetl.filters.zipfileextractor.``ZipFileExtractor`(configdict, section)

  Bases: `stetl.filter.Filter`

  Extracts a file from a ZIP file, and saves it as the given file name.

  consumes=FORMAT.record, produces=FORMAT.string

  `after_chain_invoke`(packet)

    Called right after entire Component Chain invoke.

  `delete_file`()
    `CONFIG`

      Delete the file when the chain has been completed?

    - type: bool

    - required: False

    - default: True

  `file_path`()
    `CONFIG`

      File name to write the extracted file to.

    - type: str

    - required: True

    - default: None

  `invoke`(packet)

    Components override for Component-specific behaviour, typically read, filter or write actions.

## Components: Outputs

class `stetl.outputs.fileoutput.``FileOutput`(configdict, section)

  Bases: `stetl.output.Output`

  Pretty print input to file. Input may be an etree doc or any other stringify-able input.

  consumes=FORMAT.any

  `file_path`()
    `CONFIG`

      Path to file, for MultiFileOutput can be of the form like: gmlcities-%03d.gml

    - type: str

    - required: True

    - default: None

class `stetl.outputs.fileoutput.``MultiFileOutput`(configdict, section)

  Bases: `stetl.outputs.fileoutput.FileOutput`

  Print to multiple files from subsequent packets like strings or etree docs, file_path must be of a form like: gmlcities-%03d.gml.

  consumes=FORMAT.any

class `stetl.outputs.standardoutput.``StandardOutput`(configdict, section)

  Bases: `stetl.output.Output`

  Print any input to standard output.

  consumes=FORMAT.any

class `stetl.outputs.standardoutput.``StandardXmlOutput`(configdict, section)

  Bases: `stetl.output.Output`

  Pretty print XML from etree doc to standard output. OBSOLETE, can be done with StandardOutput

  consumes=FORMAT.etree_doc

class `stetl.outputs.ogroutput.``Ogr2OgrOutput`(configdict, section)

  Bases: `stetl.output.Output`

  Output from GML etree doc to any OGR2OGR output using the GDAL/OGR ogr2ogr command

  consumes=FORMAT.etree_doc

class `stetl.outputs.ogroutput.``OgrOutput`(configdict, section)

  Bases: `stetl.output.Output`

  Direct GDAL OGR output via Python OGR wrapper. Via the Python API http://gdal.org/python OGR Features are written.

  This output can write almost any geospatial, OGR-defined, dataformat.

  consumes=FORMAT.ogr_feature or FORMAT.ogr_feature_array

  `append`()
    `CONFIG`

      Add to destination destination if it extists (ogr2ogr -append option).

    - type: bool

    - required: False

    - default: False

  `dest_create_options`()
    `CONFIG`

      Creation options.

      Examples: ..

    - type: list

    - required: False

    - default: []

  `dest_data_source`()
    `CONFIG`

      String denoting the OGR data destination. Usually a path to a file like “path/rivers.shp” or connection string to PostgreSQL like “PG: host=localhost dbname=’rivers’ user=’postgres’”.

    - type: str

    - required: True

    - default: None

  `dest_format`()
    `CONFIG`

      Instructs GDAL to use driver by that name to open data destination. Not required for many standard formats that are self-describing like ESRI Shapefile.

      Examples: ‘PostgreSQL’, ‘GeoJSON’ etc

    - type: str

    - required: False

    - default: None

  `dest_options`()
    `CONFIG`

      Custom data destination-specific options. Used in gdal.SetConfigOption().

    - type: dict

    - required: False

    - default: None

  `init`()

    Allows derived Components to perform a one-time init.

  `layer_create_options`()
    `CONFIG`

      Options for newly created layer (-lco).

    - type: list

    - required: False

    - default: []

  `new_layer_name`()
    `CONFIG`

      Layer name for layer created in the destination source.

    - type: str

    - required: True

    - default: None

  `overwrite`()
    `CONFIG`

      Overwrite destination if it extists (ogr2ogr -overwrite option).

    - type: bool

    - required: False

    - default: False

  `sql`()
    `CONFIG`

      String with SQL query. Mandatory for PostgreSQL OGR dest.

    - type: str

    - required: False

    - default: None

  `target_srs`()
    `CONFIG`

      SRS (projection) for the target.

    - type: str

    - required: False

    - default: None

class `stetl.outputs.execoutput.``CommandExecOutput`(configdict, section)

  Bases: `stetl.outputs.execoutput.ExecOutput`

  Executes an arbitrary command.

  consumes=FORMAT.string

class `stetl.outputs.execoutput.``ExecOutput`(configdict, section, consumes)

  Bases: `stetl.output.Output`

  Executes any command (abstract base class).

  `env_args`()
    `CONFIG`

      Provides of list of environment variables which will be used when executing the given command.

      Example: env_args = pgpassword=postgres othersetting=value~with~spaces

    - type: str

    - required: False

    - default:

  `env_separator`()
    `CONFIG`

      Provides the separator to split the environment variable names from their values.

    - type: str

    - required: False

    - default: =

class `stetl.outputs.execoutput.``Ogr2OgrExecOutput`(configdict, section)

  Bases: `stetl.outputs.execoutput.ExecOutput`

  Executes an Ogr2Ogr command. Input is a file name to be processed. Output by calling Ogr2Ogr command.

  consumes=FORMAT.string

  `cleanup_input`()
    `CONFIG`

      Flag to indicate whether the input file to ogr2ogr should be cleaned up.

    - type: bool

    - required: False

    - default: False

  `dest_data_source`()
    `CONFIG`

      String denoting the OGR data destination. Usually a path to a file like “path/rivers.shp” or connection string to PostgreSQL like “PG: host=localhost dbname=’rivers’ user=’postgres’”.

    - type: str

    - required: True

    - default: None

  `dest_format`()
    `CONFIG`

      Instructs GDAL to use driver by that name to open data destination. Not required for many standard formats that are self-describing like ESRI Shapefile.

      Examples: ‘PostgreSQL’, ‘GeoJSON’ etc

    - type: str

    - required: False

    - default: None

  `gfs_template`()
    `CONFIG`

      Name of GFS template file to use during loading. Passed to ogr2ogr as –config GML_GFS_TEMPLATE <name>

    - type: str

    - required: False

    - default: None

  `lco`()
    `CONFIG`

      Options for newly created layer (-lco).

    - type: str

    - required: False

    - default: None

  `options`()
    `CONFIG`

      Miscellaneous options to pass to ogr2ogr.

    - type: str

    - required: False

    - default: None

  `spatial_extent`()
    `CONFIG`

      Spatial extent (-spat), to pass as xmin ymin xmax ymax

    - type: str

    - required: False

    - default: None

class `stetl.outputs.dboutput.``DbOutput`(configdict, section, consumes)

  Bases: `stetl.output.Output`

  Output to any database (abstract base class).

class `stetl.outputs.dboutput.``PostgresDbOutput`(configdict, section)

  Bases: `stetl.outputs.dboutput.DbOutput`

  Output to PostgreSQL database. Input is an SQL string. Output by executing input SQL string.

  consumes=FORMAT.string

  `database`()
    `CONFIG`

      Database name.

    - type: str

    - required: True

    - default: None

  `host`()
    `CONFIG`

      Hostname for DB.

    - type: str

    - required: False

    - default: None

  `password`()
    `CONFIG`

      DB Password for user.

    - type: str

    - required: False

    - default: None

  `schema`()
    `CONFIG`

      Postgres schema name for DB.

    - type: str

    - required: False

    - default: public

  `user`()
    `CONFIG`

      DB User name.

    - type: str

    - required: False

    - default: None

class `stetl.outputs.dboutput.``PostgresInsertOutput`(configdict, section, consumes='record')

  Bases: `stetl.outputs.dboutput.PostgresDbOutput`

  Output by inserting a single record in a Postgres database table. Input is a Stetl record (Python dict structure) or a list of records. Creates an INSERT for Postgres to insert each single record. When the “replace” parameter is True, any existing record keyed by “key” is attempted to be UPDATEd first.

  NB a constraint is that the first and each subsequent each record needs to contain all values as an INSERT and UPDATE query template is built once for the columns in the first record.

  consumes=[FORMAT.record_array, FORMAT.record]

  `exit`()

    Allows derived Components to perform a one-time exit/cleanup.

  `init`()

    Allows derived Components to perform a one-time init.

  `key`()
    `CONFIG`

      The key column name of the table, required when replacing records.

    - type: str

    - required: False

    - default: None

  `replace`()
    `CONFIG`

      Replace record if exists?

    - type: bool

    - required: False

    - default: False

  `table`()
    `CONFIG`

      Table for inserts.

    - type: str

    - required: False

    - default: public

class `stetl.outputs.deegreeoutput.``DeegreeBlobstoreOutput`(configdict, section)

  Bases: `stetl.output.Output`

  Insert features into deegree Blobstore from an etree doc.

  consumes=FORMAT.etree_doc

  `init`()

    Allows derived Components to perform a one-time init.

class `stetl.outputs.deegreeoutput.``DeegreeFSLoaderOutput`(configdict, section)

  Bases: `stetl.output.Output`

  Insert features via deegree using deegree’s FSLoader tool from an etree doc.

  consumes=FORMAT.etree_doc