- Shiboken
- » Type System Reference

# Opaque Containers

Normally, Python containers such as `list` or `dict` are passed when calling C++ functions taking a corresponding C++ container (see container-type).

This means that for each call, the entire Python container is converted to a C++ container, which can be inefficient when for example creating plots from lists of points.

To work around this, special opaque containers can generated which wrap an underlying C++ container directly (currently implemented for `list` types). They implement the sequence protocol and can be passed to the function instead of a Python list. Manipulations like adding or removing elements can applied directly to them using the C++ container functions.

This is achieved by specifying the name and the instantiated type in the `opaque-containers` attribute of container-type or using the opaque-container element for existing container types.

A second use case are public fields of container types. In the normal case, they are converted to Python containers on read access. By a field modification, (see modify-field), it is possible to obtain an opaque container which avoids the conversion and allows for direct modification of elements.

Getters returning references can also be modified to return opaque containers. This is done by modifying the return type to the name of the opaque container (see replace-type).

The table below lists the functions supported for opaque sequence containers besides the sequence protocol (element access via index and `len()`). Both the STL and the Qt naming convention (which resembles Python’s) are supported:

Function | Description
`push_back(value)`, `append(value)` | Appends value to the sequence.
`push_front(value)`, `prepend(value)` | Prepends value to the sequence.
`clear()` | Clears the sequence.
`pop_back()`, `removeLast()` | Removes the last element.
`pop_front()`, `removeFirst()` | Removes the first element.
`reserve(size)` | For containers that support it (`std::vector`, `QList`), allocate memory for at least `size` elements, preventing reallocations.
`capacity()` | For containers that support it (`std::vector`, `QList`), return the number of elements that can be stored without reallocation.
`data()` | For containers that support it (`std::vector`, `QList`), return a buffer viewing the memory.
`constData()` | For containers that support it (`std::vector`, `QList`), return a read-only buffer viewing the memory.

Note

`std::span`, being a non-owning container, is currently replaced by a `std::vector` for argument passing. This means that an opaque container wrapping a `std::span` obtained from a function will be converted to a `std::vector` by sequence conversion when passed to a function taking a `std::span`. Opaque containers wrapping a `std::vector` can be passed without conversion. This is currently experimental and subject to change.

Following is an example on creating an opaque container named `IntVector` from std::vector<int>, and using it in Python.

We will consider three separate use cases.

Case 1 - When a Python list is passed to C++ function `TestOpaqueContainer.getVectorSum(const std::vector<int>&)` as an opaque container

    class TestOpaqueContainer
    {
    public:
        static int getVectorSum(const std::vector<int>& intVector)
        {
            return std::accumulate(intVector.begin(), intVector.end(), 0);
        }
    };

Case 2 - When we have a C++ class named `TestOpaqueContainer` with a `std::vector<int>` public variable

    class TestOpaqueContainer
    {
    public:
        std::vector<int> intVector;

    };

Case 3 - When we have a C++ class named `TestOpaqueContainer` with a `std::vector<int>` as private variable and the variable is returned by a reference through a getter.

    class TestOpaqueContainer
    {
    public:
        std::vector<int>& getIntVector()
        {
            return this->intVector;
        }

    private:
        std::vector<int> intVector;

    };

Note

Cases 2 and 3 are generally considered to be bad class design in C++. However, the purpose of these examples are rather to show the different possibilities with opaque containers in Shiboken than the class design.

In all the three cases, we want to use `intVector` in Python through an opaque-container. The first thing to do is to create the corresponding `<container-type />` attribute in the typesystem file, making Shiboken aware of the `IntVector`.

    <container-type name="std::vector" type="vector" opaque-containers="int:IntVector">
        <include file-name="vector" location="global"/>
        <conversion-rule>
            <native-to-target>
                <insert-template name="shiboken_conversion_cppsequence_to_pylist"/>
            </native-to-target>
            <target-to-native>
                <add-conversion type="PySequence">
                    <insert-template name="shiboken_conversion_pyiterable_to_cppsequentialcontainer"/>
                </add-conversion>
            </target-to-native>
        </conversion-rule>
    </container-type>

For the rest of the steps, we consider the three cases separately.

Case 1 - When a Python list is passed to a C++ function

As the next step, we create a typesystem entry for the class `TestOpaqueContainer`.

    <value-type name="TestOpaqueContainer" />

In this case, the typesystem entry is simple and the function `getVectorSum(const std::vector<int>&)` accepts `IntVector` as the parameter. This is because inherantly `IntVector` is the same as `std::vector<int>`.

Now, build the code to create the `*_wrapper.cpp` and `*.so` files which we import into Python.

Verifying the usage in Python

    >>> vector = IntVector()
    >>> vector.push_back(2)
    >>> vector.push_back(3)
    >>> len(vector)
    2
    >>> TestOpaqueContainer.getVectorSum(vector)
    vector sum is 5

Case 2 - When the variable is public

We create a typesystem entry for the class `TestOpaqueContainer`.

    <value-type name="TestOpaqueContainer">
        <modify-field name="intVector" opaque-container="yes"/>
    </value-type>

In the `<modify-field />` notice the `opaque-container="yes"`. Since the type of `intVector` is `std::vector<int>`, it picks up the `IntVector` opaque container.

Build the code to create the `*_wrapper.cpp` and `*.so` files which we import into Python.

Verifying the usage in Python

    >>> test = TestOpaqueContainer()
    >>> test
    <Universe.TestOpaqueContainer object at 0x7fe17ef30c30>
    >>> test.intVector.push_back(1)
    >>> test.intVector.append(2)
    >>> len(test.intVector)
    2
    >>> test.intVector[1]
    2
    >>> test.intVector.removeLast()
    >>> len(test.intVector)
    1

Case 3 - When the variable is private and returned by reference through a getter

Similar to the previous cases, we create a typesystem entry for the class `TestOpaqueContainer`.

    <value-type name="TestOpaqueContainer">
        <modify-function signature="getIntVector()">
            <modify-argument index="return">
                <replace-type modified-type="IntVector" />
            </modify-argument>
        </modify-function>
    </value-type>

In this case, we specify the name of the opaque container `IntVector` in the `<replace-type />` field.

Build the code to create the *_wrapper.cpp and *.so files which we import into Python.

Verifying the usage in Python

    >>> test = TestOpaqueContainer()
    >>> test
    <Universe.TestOpaqueContainer object at 0x7f62b9094c30>
    >>> vector = test.getIntVector()
    >>> vector
    <Universe.IntVector object at 0x7f62b91f7d00>
    >>> vector.push_back(1)
    >>> vector.push_back(2)
    >>> len(vector)
    2
    >>> vector[1]
    2
    >>> vector.removeLast()
    >>> len(vector)
    1

In all the three cases, if we check out the corresponding wrapper class for the module, we will see the lines

    static PyMethodDef IntVector_methods[] = {
        {"push_back", reinterpret_cast<PyCFunction>(
            ShibokenSequenceContainerPrivate<std::vector<int >>::push_back),METH_O, "push_back"},
        {"append", reinterpret_cast<PyCFunction>(
            ShibokenSequenceContainerPrivate<std::vector<int >>::push_back),METH_O, "append"},
        {"clear", reinterpret_cast<PyCFunction>(
            ShibokenSequenceContainerPrivate<std::vector<int >>::clear), METH_NOARGS, "clear"},
        {"pop_back", reinterpret_cast<PyCFunction>(
            ShibokenSequenceContainerPrivate<std::vector<int >>::pop_back), METH_NOARGS,
            "pop_back"},
        {"removeLast", reinterpret_cast<PyCFunction>(
            ShibokenSequenceContainerPrivate<std::vector<int >>::pop_back), METH_NOARGS,
            "removeLast"},
        {nullptr, nullptr, 0, nullptr} // Sentinel
    };

This means, the above mentioned methods are available to be used in Python with the `IntVector` opaque container.

Note

Plot example demonstrates an example of using an opaque container QPointList, which wraps a C++ QList<QPoint>. The corresponding typesystem file where QPointList can be found here