# 3D Rendering

The UI library provides the ability to draw hardware-accelerated 3D graphics. As for 2D rendering, this happens on a separate thread (`ui.Render`) to avoid blocking the `ui.Ui` thread.

Since a large part of the 3D library (particularly the language extensions) is implemented in Basic Storm, it is located in its own sub-package: `ui.s3d` so that applications that do not require 3D rendering do not need to load it. `s3d` is short for "Storm 3D".

The following people have greatly helped developing the 3D rendering in the UI library: Viggo Ingmarsson, Mattis Olevall, Rasmus Ring, Max Ryttersten, Alfred Lindström, Emil Jönsson, Elsa Annell

## Painters and Graphics

3D rendering interacts with the windowing system very similarly to the 2D rendering system. This is perhaps not surprising since they share much of the infrastructure for hardware-accelerated rendering. As such, to render 3D graphics in a window, it is necessary to subclass the `ui.Painter3D` class, override the `render` member, and attach the painter to a window by calling the `void painter(core.Maybe<ui.BasicPainter> to)` member in `ui.Window`. This will cause the UI library to call the `render` member whenever a new frame should be rendered. Either at a steady framerate, or whenever `repaint` is called.

The `ui.Painter3D` class has the following members:

- `core.Bool render(core.geometry.Size size, ui.Graphics3D graphics)`

  Called to render the contents of the window. `size` is the drawing size in device independent units. Return `true` to schedule a repaint for the next frame.

- `void repaint()`

  Called from Storm to schedule a repaint of the window.

- `graphics.Image captureImage(core.geometry.Size size)`

  Render a picture to a Bitmap. This is not necessarily a fast operation and is therefore not suitable for real-time rendering. Rather, it is intended to provide the ability to save a picture of a part of the screen to a file. Note that the painter needs to be attached to a window for this to work.

As can be seen from above, the `ui.Painter3D` class provides a `ui.Graphics3D` instance that can be used for 3D rendering. In contrast to `ui.Graphics` used for 2D rendering, the `ui.Graphics3D` class provides a relatively sparse public interface:

- `void fill(graphics.Color color)`

  Fill the entire surface with a solid color. Useful to clear the background color before rendering. Not done automatically, since it is not needed if we know that we will fill all pixels with other content anyway.

- `void clearDepth()`

  Clear the depth buffer. The depth buffer is automatically cleared to the far-plane for each frame.

## Pipelines

To render 3D geometry through a `ui.Graphics3D` object it is necessary to define a pipeline. This is done as follows in the language extension that the `ui.s3d` library provides to the Basic Storm language:

    use ui:s3d;

    pipeline ExamplePipeline {
        // Describe the data flow through the pipeline:
        data (Input input, uniform Float time) -> Intermediate -> Output;

        // Other members may be defined here. They become uniforms.
        Transform camera;

        // It is also possible to define arbitrary Basic Storm members. They
        // are only accessible to Storm, not to shaders.
    }

As can be seen above, defining a pipeline uses the keyword `pipeline` followed by a name. This intentionally resembles a class definition, since a pipeline behaves much like a class. The exception is that it contains a data definition, the data members become accessible as uniforms in the shaders. Any member functions are only accessible from Storm, not from shaders.

The first line inside a pipeline is special. It defines the data flow through the pipeline. The `data` keyword is followed by a set of parentheses that specify which parameters should be passed as inputs to the parameter. The syntax resembles the definition of function parameters. This is also intentional. As we shall see, the types passed here will become parameters to the `render` function of the pipeline.

There are two types of parameters. Vector inputs and uniforms. The first parameter (`Input input`) is a vector input. This means that we will give the `render` function an array of vertices of the type `Input` stored in a `VertexBuffer`. The rendering pipeline will then apply the shaders on each element individually (you can think of it as the system adding a `for` loop for you).

Uniform do not work this way. They are set once, and have the same value across all invocations of the shaders. As such, the system does not expect an array of `Float`s for the `time` parameter for example. Rather, it expects a single `Float` that will be used for all vertices. It is also possible to define uniforms as data members inside the pipeline. As we shall see, they work a little differently to uniforms defined in the parameter list.

After the input definition, there is an arrow `->` followed by another vertex type (`Intermediate`). This specifies the output type of the vertex shader and subsequently the input type of the pixel shader. Finally, there is another arrow `->` and another vertex type (`Output`). This type specifies the output vertex type for the pixel shader.

A pipeline is defined as a class that (indirectly) inherits from `ui.Pipeline`, and inherits some members from those classes. However, what is relevant is typically the following members generated by the pipeline definitions:

- For each member that is defined in the pipeline, the library generates a getter and a setter function. This makes it possible to access the members as if they are normal member variables (although the getters and setters track which members have changed, to know when data needs to be uploaded to the GPU).
- The static function `create(VertexShader, PixelShader)`. This function is used to create an instance of the pipeline. The `VertexShader` and `PixelShader` arguments refer to a vertex and a pixel shader that should be used in the pipeline. These are explained below.
- The `render(Graphics3D, DrawPrimitive, ...)` function. This function is used to render geometry using the pipeline. The first parameter (apart from the `this` parameter) is a reference to a `ui.Graphics3D` object that encapsulates the render target. The second parameter is of the type `ui.DrawPrimitive`, and determines what type of geometry to render (e.g. `ui.DrawPrimitive triangles()`).

  The remaining parameters depend on the parameters after the `data` declaration of the pipeline. For each parameter that is not marked as `uniform`, the `render` function will accept a `VertexBuffer` of the corresponding vertex type. For each parameter marked as `uniform`, the `render` function will accept a parameter of the corresponding type. For example, the `render` function for `ExamplePipeline` above has the following signature: `render(Graphics3D, DrawPrimitive, VertexBuffer<Input>, Float)`

  It is worth noting that uniforms defined as parameters to the `render` function are expected to change with each call to `render`. As such, in contrast to variables defined on the pipeline itself, these parameters are uploaded to the GPU every time, even if the same value is passed multiple times.

- There is also a version of `render` that accept an `ui.s3d.IndexBuffer` for rendering indexed geometry. This version of the `render` function has the same signature as the one mentioned above, except that the `ui.DrawPrimitive` parameter is replaced with a `ui.s3d.IndexBuffer`. The index buffer specifies which primitive to render.
- If the input vertex type is `ui.s3d.Vertex` (i.e. the vertex type used in `ui.s3d.Model`), there is also a version of `render` that accepts a `Model` instead of the the `ui.DrawPrimitive` and the vertex buffer. For example, if we replace `Input` with `ui:s3d:Vertex` in `ExamplePipeline`, the following `render` overload would also be available: `render(Graphics3D, Model, Float)`.

## Vertices

The pipeline requires that all inputs, outputs, and uniforms are vertices. In essence, a vertex type is a type that can be manipulated on the GPU. The name stems from the fact that they are typically used to represent vertex data in a 3D mesh (e.g. position, normal, etc.).

The following built-in types are considered primitive vertex types:

- `core.Bool`: booleans.
- `core.Int`: signed integers.
- `core.Nat`: unsigned integers.
- `core.Float`: floating point numbers.
- `core.geometry.Angle`: 1D float, known to represent an angle.
- `core.geometry.Point`: 2D vectors of floats.
- `core.geometry.Size`: 2D vectors of floats.
- `core.geometry.Vector`: 3D vectors of floats.
- `core.geometry.Vector4`: 3D vectors of floats, homogenous coordinates to allow representing translations and projections as matrices. The 3D rendering pipeline requires homogenous coordinates in some places. Otherwise it is more or less interchangeable with non-homogenous coordinates.
- `core.geometry.Rotation`: 4D quaternions used to represent arbitrary rotations in 3D.
- `core.geometry.Transform`: 4x4 matrices to represent transformations in 3D.
- `graphics.Color`: 4D vector representing a color (R, G, B, and alpha).

It is also possible to define custom composite vertex types using the `vertex` keyword. As can be seen below, vertex types are similar to class and value definitions. However, there are some limitations, most importantly that all members need to be vertex types themselves.

For example, the input type from the pipeline above may be defined as follows:

    use ui:s3d;

    vertex Input {
        Vector pos;
        Color color;
    }

At the time of writing, it is not possible to define member functions in vertex types. This may be added in the future. However, as the shader language follows the same lookup rules as Basic Storm, it is possible to call free functions as if they are member functions.

Vertex types automatically generate a set of constructors, both for the shader language and for Basic Storm. These constructors simply accept one parameter for each member of the vertex type and initialize members one by one. There are also constructors with fewer parameters than members. These simply initialize the remaining members to zero.

For example, the expression `Input(Vector(1.0, 1.0, 1.0), Color(0.0, 1.0, 0.0))` initializes both `pos` and `color`. It is also possible to write `Input(Vector(1.0, 1.0, 1.0))` to initialize `pos` as specified and `color` to zero (i.e. transparency).

Vertex types also allow marking members based on how they should be used by the rendering pipeline (DirectX call these "semantics"). This is used to (1) use the output from the vertex shader to determine which pixels need to be rendered, (2) determine how to interpolate vertices from the vertex shader as inputs to the pixel shader, and (3) determine which output from the pixel shader represents the color on the screen.

To fullfill (1) and (2) above, the vertex type that is used as an output from the vertex shader has the following requirements:

- One `core.geometry.Vector4` needs to be marked as `position`. It should contain the screen-space coordinates of the vertex. The rendering pipeline will then use this value to determine which pixels need to be rendered.
- All members of the vertex type will be interpolated according to an interpolation mode. If nothing else is specified, `perspective` is used by default. This can be changed by specifying any other supported interpolation mode.

The following interpolation modes are supported:

- `perspective`: Apply perspective correction to the interpolation.
- `linear`: Apply linear interpolation without perspective correction.
- `none`: No interpolation.

For example, the intermediate vertex can be defined as follows:

    use ui:s3d;

    vertex Intermediate {
        Vector4 pos : position, perspective;
        Color color : perspective;
    }

As can be seen above, `pos` is marked as both `position` and `perspective` for clarity. The `color` member is only marked as `perspective` for clarity. Again, this is optional as `perspective` is the default interpolation.

To fullfill (3) above, the output from the pixel shader needs to have at least one `graphics.Color` member marked `color` or `output` to let the rendering pipeline know which member to paint onto the screen. It is also possible to modify the depth value of the pixel by annotating a member with the `depth` flag (a `core.Float`).

As such, the `Output` vertex type from above can be defined as below:

    use ui:s3d;

    vertex Output {
        Color out : color;
    }

## Shaders

Shaders are defined using a syntax that is similar to ordinary functions in Basic Storm. The main difference is that shaders need to know which pipeline they belong to, so that the system knows which uniforms are available (i.e. shaders are similar to member functions, except for the fact that they are typically defined outside of the pieline itself). Shaders also need to know if they are used as a vertex shader or a pixel shader to check that inputs and outputs match the specification in the pipeline.

All of this is done using a set of keywords before a normal function definition. Vertex shaders start with `vertex shader for <pipeline>:` and pixel shaders start with `pixel shader for <pipeline>:`. In both cases, `<pipeline>` is the name of a pipeline.

The remainder of the shader definition follows the semantics of a Basic Storm function definition. That is: `<output> <name>(<parameters>)`, where `<output>` is the name of the output type, `<name>` is the name of the shader, and `<parameters>` is a comma-separated list of parameters to the shader.

Note that the output and parameter list need to match the `data` declaration in the associated pipeline. For vertex shaders, this means that the parameter list needs to match the types (but not names) in the parentheses after `data`. However, any uniform parameters should not be listed to the shader. They are available anyway by virtue of being uniforms. The output type must match the type after the first arrow (`->`).

For the pixel shader, the input must match the type after the first arrow (`->`). There may be additional inputs that correspond to quantities that are generated by the system (e.g. current Z depth, but this is not yet implemented). The output type must match the type after the last arrow (`->`) in the pipeline.

The code in the body of the shader is written in a reduced version of Basic Storm. In general, operations that can be used with the vertex types above are supported, basic control flow (`if`, `for`, `while`), and `return`, but not much else. The shader code is then compiled to GLSL or HLSL depending on which rendering system is used on the target platform.

Below are two example shader functions that can be used with the `ExamplePipeline` defined earlier. Note that the uniforms (e.g. `camera`) can be accessed as if they are global variables.

    use ui:s3d;

    vertex shader for ExamplePipeline: Intermediate vertexShader(Input input) {
        Vector4 pos = Vector4(input.pos) * camera;
        Intermediate(pos, input.color);
    }

    pixel shader for ExamplePipeline: Output pixelShader(Intermediate point) {
        Output(point.color);
    }

Shader functions then become available in the name tree. They are exposed as functions that simply return a corresponding shader object. As such, they can be used to create a pipeline as shown below.

    SimplePipeline pipeline = SimplePipeline:create(vertexShader, pixelShader);

It is also possible to define functions that are usable by shaders (shader functions). These differ from vertex and pixel shaders in that they are not associated with any particular pipeline. They are therefore usable from any shader from any pipeline. However, this also means that it is not possible to access uniforms from within shader functions, unless they are passed as parameters to the shader function.

Shader functions are declared by prepending the keyword `shader` before a Basic Storm function definition. For example, a function that mixes two colors together may be defined as follows:

    use ui:s3d;

    shader Color lerp(Color a, Color b, Float t) {
        return a * t + b * (1 - t);
    }

Note: it is currently not possible to call neither shaders nor shader functions directly from Basic Storm. This may be possible in the future to simplify testing.

## Textures

Textures in the 3D rendering process are represented by the `ui.Texture` class. Instances of this class represent a texture that is accessible to the GPU, and the associated sampler state (e.g. how to handle wrapping, mip-mapping, etc.).

Even though textures are not vertex types, it is possible to define a texture variable in a pipeline. The pipeline is aware of the `ui.Texture` class and handles it appropriately.

The `ui.Texture` class has the following members:

- `graphics.Image image()`

  Get the current image.

- `void image(graphics.Image newImage)`

  Set the current image. Note, this causes the image to be uploaded to the texture again. If you wish to render multiple textures on different passes, it is better to create different `Texture` instances and assign them to the pipeline, rather than switching the image in the same `Texture` object multiple times.

- `graphics.Color sample(core.geometry.Point coords)`

  Sample the image. This member can be used in shaders to sample the texture.

In particular, note that `Texture` objects are uploaded to the GPU on demand. Replacing the image inside a texture always causes the new image to be uploaded to the GPU. As such, if you are using one pipeline to render multiple textures in a single frame, it is better to create multiple `Texture` objects and simply assign different ones to the shader rather than replacing the image in the texture.

The `graphics.Color sample(core.geometry.Point coords)` function can be called both on the CPU and the GPU to sample textures. As such, applying a texture to geometry can be done with a pixel shader that looks like below (assumes that `Intermediate` has a `tex` member that stores texture coordinates, and that the pipeline has a texture named `texture`).

    use ui:s3d;

    pixel shader for TexturePipeline: Output pixelShader(Intermediate point) {
        Color texColor = texture.sample(point.tex);
        Output(texColor);
    }

## Models

The 3D library contains the class `ui.s3d.Model`, which represents some 3D model that can be rendered. In essence, it wraps an index buffer and a vertex buffer together to provide a standardized representation for exchanging geometry and some rudimentary operations.

The `ui.s3d.Model` class has the following members:

- `ui.s3d.VertexBuffer<ui.s3d.Vertex> vertices`

  Vertex buffer containing all vertices in the model.

- `ui.s3d.IndexBuffer indices`

  Index buffer containing the indices in the model.

- `void append(ui.s3d.Model n)`

  Append another model to this one.

- `ui.s3d.Model transform(core.geometry.Transform tfm)`

  Translate the model. Return the object itself to allow chaining things.

- `void color(graphics.Color c)`

  Set the color for all vertices (assign function).

- `ui.s3d.Model withColor(graphics.Color c)`

  Set the color of this model, and return it to allow chaining operations.

It also provides the following static member functions to create some standard 3D shapes:

- `ui.s3d.Model cube(core.Float size)`

  Create a cube with the specified size centered at 0, 0, 0.

- `ui.s3d.Model unitCube()`

  Create a unit cube centered at 0, 0, 0.

- `ui.s3d.Model sphere(core.Float radius, core.Nat sectors, core.Nat stacks)`

  Create a sphere centered at 0, 0, 0.

- `ui.s3d.Model unitSphere(core.Nat sectors, core.Nat stacks)`

  Create a unit sphere centered at 0, 0, 0.

It is also possible to load 3D models from `.obj` files using the `loadObj` function:

- `ui.s3d.Model loadOBJ(core.io.Url file)`

  Load 3D object from an `.obj` file.

- `ui.s3d.Model loadOBJ(core.io.IStream from)`

  Load a 3D object from an `.obj` file.

The model uses the vertex type `ui.s3d.Vertex`, which has the following members:

- `core.geometry.Vector pos`

  Vertex position.

- `core.geometry.Vector normal`

  Vertex normal. Assumed to be normalized to length 1.

- `graphics.Color color`

  Vertex color.

- `core.geometry.Point tex`

  Texture coordinate. Each coordinate is normally between 0 and 1.

## Example usage

The tutorials section of the documentation contains a few examples of how to use the 3D part of the UI library to render some simple geometry and apply some simple effects.