# `pwnlib.util.packing` — Packing and unpacking of strings

Module for packing and unpacking integers.

Simplifies access to the standard `struct.pack` and `struct.unpack` functions, and also adds support for packing/unpacking arbitrary-width integers.

The packers are all context-aware for `endian` and `signed` arguments, though they can be overridden in the parameters.

Examples

    >>> p8(0)
    b'\x00'
    >>> p32(0xdeadbeef)
    b'\xef\xbe\xad\xde'
    >>> p32(0xdeadbeef, endian='big')
    b'\xde\xad\xbe\xef'
    >>> with context.local(endian='big'): p32(0xdeadbeef)
    b'\xde\xad\xbe\xef'

Make a frozen packer, which does not change with context.

    >>> p=make_packer('all')
    >>> p(0xff)
    b'\xff'
    >>> p(0x1ff)
    b'\xff\x01'
    >>> with context.local(endian='big'): print(repr(p(0x1ff)))
    b'\xff\x01'

pwnlib.util.packing.dd(dst, src, count=0, skip=0, seek=0, truncate=False) → dst

  Inspired by the command line tool `dd`, this function copies count byte values from offset seek in src to offset skip in dst. If count is 0, all of `src[seek:]` is copied.

  If dst is a mutable type it will be updated. Otherwise, a new instance of the same type will be created. In either case the result is returned.

  src can be an iterable of characters or integers, a unicode string or a file object. If it is an iterable of integers, each integer must be in the range [0;255]. If it is a unicode string, its UTF-8 encoding will be used.

  The seek offset of file objects will be preserved.

  Parameters:

    - dst – Supported types are `file`, `list`, `tuple`, `str`, `bytearray` and `unicode`.

    - src – An iterable of byte values (characters or integers), a unicode string or a file object.

    - count (int) – How many bytes to copy. If count is 0 or larger than `len(src[seek:])`, all bytes until the end of src are copied.

    - skip (int) – Offset in dst to copy to.

    - seek (int) – Offset in src to copy from.

    - truncate (bool) – If `True`, dst is truncated at the last copied byte.

  Returns:

    A modified version of dst. If dst is a mutable type it will be modified in-place.

  Examples

      >>> dd(tuple('Hello!'), b'?', skip = 5)
      ('H', 'e', 'l', 'l', 'o', b'?')
      >>> dd(list('Hello!'), (63,), skip = 5)
      ['H', 'e', 'l', 'l', 'o', b'?']
      >>> _ = open('/tmp/foo', 'w').write('A' * 10)
      >>> dd(open('/tmp/foo'), open('/dev/zero'), skip = 3, count = 4).read()
      'AAA\x00\x00\x00\x00AAA'
      >>> _ = open('/tmp/foo', 'w').write('A' * 10)
      >>> dd(open('/tmp/foo'), open('/dev/zero'), skip = 3, count = 4, truncate = True).read()
      'AAA\x00\x00\x00\x00'

pwnlib.util.packing.fit(*args, **kwargs)

  Legacy alias for `flat()`

pwnlib.util.packing.flat(*args, **kwargs)
  flat(*args, preprocessor = None, length = None, filler = de_bruijn(),

    word_size = None, endianness = None, sign = None) -> str

  Flattens the arguments into a string.

  This function takes an arbitrary number of arbitrarily nested lists, tuples and dictionaries. It will then find every string and number inside those and flatten them out. Strings are inserted directly while numbers are packed using the `pack()` function. Unicode strings are UTF-8 encoded.

  Dictionary keys give offsets at which to place the corresponding values (which are recursively flattened). Offsets are relative to where the flattened dictionary occurs in the output (i.e. `{0: 'foo'}` is equivalent to `'foo'`). Offsets can be integers, unicode strings or regular strings. Integer offsets >= `2**(word_size-8)` are converted to a string using `pack()`. Unicode strings are UTF-8 encoded. After these conversions offsets are either integers or strings. In the latter case, the offset will be the lowest index at which the string occurs in filler. See examples below.

  Space between pieces of data is filled out using the iterable filler. The n’th byte in the output will be byte at index `n % len(iterable)` byte in filler if it has finite length or the byte at index n otherwise.

  If length is given, the output will be padded with bytes from filler to be this size. If the output is longer than length, a `ValueError` exception is raised.

  The three kwargs word_size, endianness and sign will default to using values in `pwnlib.context` if not specified as an argument.

  Parameters:

    - args – Values to flatten

    - preprocessor (function) – Gets called on every element to optionally transform the element before flattening. If `None` is returned, then the original value is used.

    - length – The length of the output.

    - filler – Iterable to use for padding.

    - word_size (int) – Word size of the converted integer.

    - endianness (str) – Endianness of the converted integer (“little”/”big”).

    - sign (str) – Signedness of the converted integer (False/True)

  Examples

  (Test setup, please ignore)

      >>> context.clear()

  Basic usage of `flat()` works similar to the pack() routines.

      >>> flat(4)
      b'\x04\x00\x00\x00'

  `flat()` works with strings, bytes, lists, and dictionaries.

      >>> flat(b'X')
      b'X'
      >>> flat([1,2,3])
      b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00'
      >>> flat({4:b'X'})
      b'aaaaX'

  `flat()` flattens all of the values provided, and allows nested lists and dictionaries.

      >>> flat([{4:b'X'}] * 2)
      b'aaaaXaaacX'
      >>> flat([[[[[[[[[1]]]], 2]]]]])
      b'\x01\x00\x00\x00\x02\x00\x00\x00'

  You can also provide additional arguments like endianness, word-size, and whether the values are treated as signed or not.

      >>> flat(1, b"test", [[[b"AB"]*2]*3], endianness = 'little', word_size = 16, sign = False)
      b'\x01\x00testABABABABABAB'

  A preprocessor function can be provided in order to modify the values in-flight. This example converts increments each value by 1, then converts to a byte string.

      >>> flat([1, [2, 3]], preprocessor = lambda x: str(x+1).encode())
      b'234'

  Using dictionaries is a fast way to get specific values at specific offsets, without having to do `data += "foo"` repeatedly.

      >>> flat({12: 0x41414141,
      ...       24: b'Hello',
      ...      })
      b'aaaabaaacaaaAAAAeaaafaaaHello'

  Dictionary usage permits directly using values derived from `cyclic()`. See `cyclic()`, :function:`pwnlib.context.context.cyclic_alphabet`, and `context.cyclic_size` for more options.

  The cyclic pattern can be provided as either the text or hexadecimal offset.

      >>> flat({ 0x61616162: b'X'})
      b'aaaaX'
      >>> flat({'baaa': b'X'})
      b'aaaaX'

  Fields do not have to be in linear order, and can be freely mixed. This also works with cyclic offsets.

      >>> flat({2: b'A', 0:b'B'})
      b'BaA'
      >>> flat({0x61616161: b'x', 0x61616162: b'y'})
      b'xaaay'
      >>> flat({0x61616162: b'y', 0x61616161: b'x'})
      b'xaaay'

  Fields do not have to be in order, and can be freely mixed.

      >>> flat({'caaa': b'XXXX', 16: b'\x41', 20: 0xdeadbeef})
      b'aaaabaaaXXXXdaaaAaaa\xef\xbe\xad\xde'
      >>> flat({ 8: [0x41414141, 0x42424242], 20: b'CCCC'})
      b'aaaabaaaAAAABBBBeaaaCCCC'
      >>> fit({
      ...     0x61616161: b'a',
      ...     1: b'b',
      ...     0x61616161+2: b'c',
      ...     3: b'd',
      ... })
      b'abadbaaac'

  By default, gaps in the data are filled in with the `cyclic()` pattern. You can customize this by providing an iterable or method for the `filler` argument.

      >>> flat({12: b'XXXX'}, filler = b'_', length = 20)
      b'____________XXXX____'
      >>> flat({12: b'XXXX'}, filler = b'AB', length = 20)
      b'ABABABABABABXXXXABAB'

  Nested dictionaries also work as expected.

      >>> flat({4: {0: b'X', 4: b'Y'}})
      b'aaaaXaaaY'
      >>> fit({4: {4: b'XXXX'}})
      b'aaaabaaaXXXX'

  Negative indices are also supported, though this only works for integer keys.

      >>> flat({-4: b'x', -1: b'A', 0: b'0', 4: b'y'})
      b'xaaA0aaay'

pwnlib.util.packing.make_packer(word_size=None, endianness=None, sign=None) → number → str

  Creates a packer by “freezing” the given arguments.

  Semantically calling `make_packer(w, e, s)(data)` is equivalent to calling `pack(data, w, e, s)`. If word_size is one of 8, 16, 32 or 64, it is however faster to call this function, since it will then use a specialized version.

  Parameters:

    - word_size (int) – The word size to be baked into the returned packer or the string all (in bits).

    - endianness (str) – The endianness to be baked into the returned packer. (“little”/”big”)

    - sign (str) – The signness to be baked into the returned packer. (“unsigned”/”signed”)

    - kwargs – Additional context flags, for setting by alias (e.g. `endian=` rather than index)

  Returns:

    A function, which takes a single argument in the form of a number and returns a string of that number in a packed form.

  Examples

      >>> p = make_packer(32, endian='little', sign='unsigned')
      >>> p
      <function _p32lu at 0x...>
      >>> p(42)
      b'*\x00\x00\x00'
      >>> p(-1)
      Traceback (most recent call last):
          ...
      error: integer out of range for 'I' format code
      >>> make_packer(33, endian='little', sign='unsigned')
      <function ...<lambda> at 0x...>

pwnlib.util.packing.make_unpacker(word_size=None, endianness=None, sign=None, **kwargs) → str → number

  Creates an unpacker by “freezing” the given arguments.

  Semantically calling `make_unpacker(w, e, s)(data)` is equivalent to calling `unpack(data, w, e, s)`. If word_size is one of 8, 16, 32 or 64, it is however faster to call this function, since it will then use a specialized version.

  Parameters:

    - word_size (int) – The word size to be baked into the returned packer (in bits).

    - endianness (str) – The endianness to be baked into the returned packer. (“little”/”big”)

    - sign (str) – The signness to be baked into the returned packer. (“unsigned”/”signed”)

    - kwargs – Additional context flags, for setting by alias (e.g. `endian=` rather than index)

  Returns:

    A function, which takes a single argument in the form of a string and returns a number of that string in an unpacked form.

  Examples

      >>> u = make_unpacker(32, endian='little', sign='unsigned')
      >>> u
      <function _u32lu at 0x...>
      >>> hex(u(b'/bin'))
      '0x6e69622f'
      >>> u(b'abcde')
      Traceback (most recent call last):
          ...
      error: unpack requires a string argument of length 4
      >>> make_unpacker(33, endian='little', sign='unsigned')
      <function ...<lambda> at 0x...>

pwnlib.util.packing.p16(number, sign, endian, ...) → bytes

  Packs an 16-bit integer

  Parameters:

    - number (int) – Number to convert

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (“unsigned”/”signed”)

    - kwargs (dict) – Arguments passed to context.local(), such as `endian` or `signed`.

  Returns:

    The packed number as a byte string

pwnlib.util.packing.p32(number, sign, endian, ...) → bytes

  Packs an 32-bit integer

  Parameters:

    - number (int) – Number to convert

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (“unsigned”/”signed”)

    - kwargs (dict) – Arguments passed to context.local(), such as `endian` or `signed`.

  Returns:

    The packed number as a byte string

pwnlib.util.packing.p64(number, sign, endian, ...) → bytes

  Packs an 64-bit integer

  Parameters:

    - number (int) – Number to convert

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (“unsigned”/”signed”)

    - kwargs (dict) – Arguments passed to context.local(), such as `endian` or `signed`.

  Returns:

    The packed number as a byte string

pwnlib.util.packing.p8(number, sign, endian, ...) → bytes

  Packs an 8-bit integer

  Parameters:

    - number (int) – Number to convert

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (“unsigned”/”signed”)

    - kwargs (dict) – Arguments passed to context.local(), such as `endian` or `signed`.

  Returns:

    The packed number as a byte string

pwnlib.util.packing.pack(number, word_size=None, endianness=None, sign=None, **kwargs) → str

  Packs arbitrary-sized integer.

  Word-size, endianness and signedness is done according to context.

  word_size can be any positive number or the string “all”. Choosing the string “all” will output a string long enough to contain all the significant bits and thus be decodable by `unpack()`.

  word_size can be any positive number. The output will contain word_size/8 rounded up number of bytes. If word_size is not a multiple of 8, it will be padded with zeroes up to a byte boundary.

  Parameters:

    - number (int) – Number to convert

    - word_size (int) – Word size of the converted integer or the string ‘all’ (in bits).

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (False/True)

    - kwargs – Anything that can be passed to context.local

  Returns:

    The packed number as a string.

  Examples

      >>> pack(0x414243, 24, 'big', True)
      b'ABC'
      >>> pack(0x414243, 24, 'little', True)
      b'CBA'
      >>> pack(0x814243, 24, 'big', False)
      b'\x81BC'
      >>> pack(0x814243, 24, 'big', True)
      Traceback (most recent call last):
         ...
      ValueError: pack(): number does not fit within word_size
      >>> pack(0x814243, 25, 'big', True)
      b'\x00\x81BC'
      >>> pack(-1, 'all', 'little', True)
      b'\xff'
      >>> pack(-256, 'all', 'big', True)
      b'\xff\x00'
      >>> pack(0x0102030405, 'all', 'little', True)
      b'\x05\x04\x03\x02\x01'
      >>> pack(-1)
      b'\xff\xff\xff\xff'
      >>> pack(0x80000000, 'all', 'big', True)
      b'\x00\x80\x00\x00\x00'

pwnlib.util.packing.u16(number, sign, endian, ...) → int

  Unpacks an 16-bit integer

  Parameters:

    - data (bytes) – Byte string to convert

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (“unsigned”/”signed”)

    - kwargs (dict) – Arguments passed to context.local(), such as `endian` or `signed`.

  Returns:

    The unpacked number

pwnlib.util.packing.u32(number, sign, endian, ...) → int

  Unpacks an 32-bit integer

  Parameters:

    - data (bytes) – Byte string to convert

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (“unsigned”/”signed”)

    - kwargs (dict) – Arguments passed to context.local(), such as `endian` or `signed`.

  Returns:

    The unpacked number

pwnlib.util.packing.u64(number, sign, endian, ...) → int

  Unpacks an 64-bit integer

  Parameters:

    - data (bytes) – Byte string to convert

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (“unsigned”/”signed”)

    - kwargs (dict) – Arguments passed to context.local(), such as `endian` or `signed`.

  Returns:

    The unpacked number

pwnlib.util.packing.u8(number, sign, endian, ...) → int

  Unpacks an 8-bit integer

  Parameters:

    - data (bytes) – Byte string to convert

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (“unsigned”/”signed”)

    - kwargs (dict) – Arguments passed to context.local(), such as `endian` or `signed`.

  Returns:

    The unpacked number

pwnlib.util.packing.unpack(data, word_size=None, endianness=None, sign=None, **kwargs) → int

  Unpacks arbitrary-sized integer.

  Word-size, endianness and signedness is done according to context.

  word_size can be any positive number or the string “all”. Choosing the string “all” is equivalent to `len(data)*8`.

  If word_size is not a multiple of 8, then the bits used for padding are discarded.

  Parameters:

    - number (int) – String to convert

    - word_size (int) – Word size of the converted integer or the string “all” (in bits).

    - endianness (str) – Endianness of the converted integer (“little”/”big”)

    - sign (str) – Signedness of the converted integer (False/True)

    - kwargs – Anything that can be passed to context.local

  Returns:

    The unpacked number.

  Examples

      >>> hex(unpack(b'\xaa\x55', 16, endian='little', sign=False))
      '0x55aa'
      >>> hex(unpack(b'\xaa\x55', 16, endian='big', sign=False))
      '0xaa55'
      >>> hex(unpack(b'\xaa\x55', 16, endian='big', sign=True))
      '-0x55ab'
      >>> hex(unpack(b'\xaa\x55', 15, endian='big', sign=True))
      '0x2a55'
      >>> hex(unpack(b'\xff\x02\x03', 'all', endian='little', sign=True))
      '0x302ff'
      >>> hex(unpack(b'\xff\x02\x03', 'all', endian='big', sign=True))
      '-0xfdfd'

pwnlib.util.packing.unpack_many(data, word_size=None, endianness=None, sign=None) → int list

  Splits data into groups of `word_size//8` bytes and calls `unpack()` on each group. Returns a list of the results.

  word_size must be a multiple of 8 or the string “all”. In the latter case a singleton list will always be returned.

  Args

    number (int): String to convert word_size (int): Word size of the converted integers or the string “all” (in bits). endianness (str): Endianness of the converted integer (“little”/”big”) sign (str): Signedness of the converted integer (False/True) kwargs: Anything that can be passed to context.local

  Returns:

    The unpacked numbers.

  Examples

      >>> list(map(hex, unpack_many(b'\xaa\x55\xcc\x33', 16, endian='little', sign=False)))
      ['0x55aa', '0x33cc']
      >>> list(map(hex, unpack_many(b'\xaa\x55\xcc\x33', 16, endian='big', sign=False)))
      ['0xaa55', '0xcc33']
      >>> list(map(hex, unpack_many(b'\xaa\x55\xcc\x33', 16, endian='big', sign=True)))
      ['-0x55ab', '-0x33cd']
      >>> list(map(hex, unpack_many(b'\xff\x02\x03', 'all', endian='little', sign=True)))
      ['0x302ff']
      >>> list(map(hex, unpack_many(b'\xff\x02\x03', 'all', endian='big', sign=True)))
      ['-0xfdfd']