# Qgram Tokenizer

class py_stringmatching.tokenizer.qgram_tokenizer.QgramTokenizer(qval=2, padding=True, prefix_pad='#', suffix_pad='$', return_set=False)

  Returns tokens that are sequences of q consecutive characters.

  A qgram of an input string s is a substring t (of s) which is a sequence of q consecutive characters. Qgrams are also known as ngrams or kgrams.

  Parameters:

    - qval (int) – A value for q, that is, the qgram’s length (defaults to 2).

    - return_set (boolean) – A flag to indicate whether to return a set of tokens or a bag of tokens (defaults to False).

    - padding (boolean) – A flag to indicate whether a prefix and a suffix should be added to the input string (defaults to True).

    - prefix_pad (str) – A character (that is, a string of length 1 in Python) that should be replicated (qval-1) times and prepended to the input string, if padding was set to True (defaults to ‘#’).

    - suffix_pad (str) – A character (that is, a string of length 1 in Python) that should be replicated (qval-1) times and appended to the input string, if padding was set to True (defaults to ‘$’).

  qval

    An attribute to store the q value.

    Type:

      int

  return_set

    An attribute to store the flag return_set.

    Type:

      boolean

  padding

    An attribute to store the padding flag.

    Type:

      boolean

  prefix_pad

    An attribute to store the prefix string that should be used for padding.

    Type:

      str

  suffix_pad

    An attribute to store the suffix string that should be used for padding.

    Type:

      str

  get_padding()

    Gets the value of the padding flag. This flag determines whether the padding should be done for the input strings or not.

    Returns:

      The Boolean value of the padding flag.

  get_prefix_pad()

    Gets the value of the prefix pad.

    Returns:

      The prefix pad string.

  get_qval()

    Gets the value of the qval attribute, which is the length of qgrams.

    Returns:

      The value of the qval attribute.

  get_return_set()

    Gets the value of the return_set flag.

    Returns:

      The boolean value of the return_set flag.

  get_suffix_pad()

    Gets the value of the suffix pad.

    Returns:

      The suffix pad string.

  set_padding(padding)

    Sets the value of the padding flag.

    Parameters:

      padding (boolean) – Flag to indicate whether padding should be done or not.

    Returns:

      The Boolean value of True is returned if the update was successful.

    Raises:

      AssertionError – If the padding is not of type boolean

  set_prefix_pad(prefix_pad)

    Sets the value of the prefix pad string.

    Parameters:

      prefix_pad (str) – String that should be prepended to the input string before tokenization.

    Returns:

      The Boolean value of True is returned if the update was successful.

    Raises:

      - AssertionError – If the prefix_pad is not of type string.

      - AssertionError – If the length of prefix_pad is not one.

  set_qval(qval)

    Sets the value of the qval attribute.

    Parameters:

      qval (int) – A value for q (the length of qgrams).

    Raises:

      AssertionError – If qval is less than 1.

  set_return_set(return_set)

    Sets the value of the return_set flag.

    Parameters:

      return_set (boolean) – a flag to indicate whether to return a set of tokens instead of a bag of tokens.

  set_suffix_pad(suffix_pad)

    Sets the value of the suffix pad string.

    Parameters:

      suffix_pad (str) – String that should be appended to the input string before tokenization.

    Returns:

      The boolean value of True is returned if the update was successful.

    Raises:

      - AssertionError – If the suffix_pad is not of type string.

      - AssertionError – If the length of suffix_pad is not one.

  tokenize(input_string)

    Tokenizes input string into qgrams.

    Parameters:

      input_string (str) – The string to be tokenized.

    Returns:

      A Python list, which is a set or a bag of qgrams, depending on whether return_set flag is True or False.

    Raises:

      TypeError – If the input is not a string

    Examples

        >>> qg2_tok = QgramTokenizer()
        >>> qg2_tok.tokenize('database')
        ['#d', 'da', 'at', 'ta', 'ab', 'ba', 'as', 'se', 'e$']
        >>> qg2_tok.tokenize('a')
        ['#a', 'a$']
        >>> qg3_tok = QgramTokenizer(qval=3)
        >>> qg3_tok.tokenize('database')
        ['##d', '#da', 'dat', 'ata', 'tab', 'aba', 'bas', 'ase', 'se$', 'e$$']
        >>> qg3_nopad = QgramTokenizer(padding=False)
        >>> qg3_nopad.tokenize('database')
        ['da', 'at', 'ta', 'ab', 'ba', 'as', 'se']
        >>> qg3_diffpads = QgramTokenizer(prefix_pad='^', suffix_pad='!')
        >>> qg3_diffpads.tokenize('database')
        ['^d', 'da', 'at', 'ta', 'ab', 'ba', 'as', 'se', 'e!']