# Utilities

This chapter catalogues tools and techniques which are useful for common tasks. The libev man page already covers some patterns which can be adopted to libuv through simple API changes. It also covers parts of the libuv API that don’t require entire chapters dedicated to them.

## Timers

Timers invoke the callback after a certain time has elapsed since the timer was started. libuv timers can also be set to invoke at regular intervals instead of just once.

Simple use is to init a watcher and start it with a `timeout`, and optional `repeat`. Timers can be stopped at any time.

    uv_timer_t timer_req;

    uv_timer_init(loop, &timer_req);
    uv_timer_start(&timer_req, callback, 5000, 2000);

will start a repeating timer, which first starts 5 seconds (the `timeout`) after the execution of `uv_timer_start`, then repeats every 2 seconds (the `repeat`). Use:

    uv_timer_stop(&timer_req);

to stop the timer. This can be used safely from within the callback as well.

The repeat interval can be modified at any time with:

    uv_timer_set_repeat(uv_timer_t *timer, int64_t repeat);

which will take effect when possible. If this function is called from a timer callback, it means:

- If the timer was non-repeating, the timer has already been stopped. Use `uv_timer_start` again.

- If the timer is repeating, the next timeout has already been scheduled, so the old repeat interval will be used once more before the timer switches to the new interval.

The utility function:

    int uv_timer_again(uv_timer_t *)

applies only to repeating timers and is equivalent to stopping the timer and then starting it with both initial `timeout` and `repeat` set to the old `repeat` value. If the timer hasn’t been started it fails (error code `UV_EINVAL`) and returns -1.

An actual timer example is in the reference count section.

## Event loop reference count

The event loop only runs as long as there are active handles. This system works by having every handle increase the reference count of the event loop when it is started and decreasing the reference count when stopped. It is also possible to manually change the reference count of handles using:

    void uv_ref(uv_handle_t*);
    void uv_unref(uv_handle_t*);

These functions can be used to allow a loop to exit even when a watcher is active or to use custom objects to keep the loop alive.

The latter can be used with interval timers. You might have a garbage collector which runs every X seconds, or your network service might send a heartbeat to others periodically, but you don’t want to have to stop them along all clean exit paths or error scenarios. Or you want the program to exit when all your other watchers are done. In that case just unref the timer immediately after creation so that if it is the only watcher running then `uv_run` will still exit.

This is also used in node.js where some libuv methods are being bubbled up to the JS API. A `uv_handle_t` (the superclass of all watchers) is created per JS object and can be ref/unrefed.

ref-timer/main.c

     1uv_loop_t *loop;
     2uv_timer_t gc_req;
     3uv_timer_t fake_job_req;
     4
     5int main() {
     6    loop = uv_default_loop();
     7
     8    uv_timer_init(loop, &gc_req);
     9    uv_unref((uv_handle_t*) &gc_req);
    10
    11    uv_timer_start(&gc_req, gc, 0, 2000);
    12
    13    // could actually be a TCP download or something
    14    uv_timer_init(loop, &fake_job_req);
    15    uv_timer_start(&fake_job_req, fake_job, 9000, 0);
    16    return uv_run(loop, UV_RUN_DEFAULT);
    17}

We initialize the garbage collector timer, then immediately `unref` it. Observe how after 9 seconds, when the fake job is done, the program automatically exits, even though the garbage collector is still running.

## Idler pattern

The callbacks of idle handles are invoked once per event loop. The idle callback can be used to perform some very low priority activity. For example, you could dispatch a summary of the daily application performance to the developers for analysis during periods of idleness, or use the application’s CPU time to perform SETI calculations :) An idle watcher is also useful in a GUI application. Say you are using an event loop for a file download. If the TCP socket is still being established and no other events are present your event loop will pause (block), which means your progress bar will freeze and the user will face an unresponsive application. In such a case queue up and idle watcher to keep the UI operational.

idle-compute/main.c

     1uv_loop_t *loop;
     2uv_fs_t stdin_watcher;
     3uv_idle_t idler;
     4char buffer[1024];
     5
     6int main() {
     7    loop = uv_default_loop();
     8
     9    uv_idle_init(loop, &idler);
    10
    11    uv_buf_t buf = uv_buf_init(buffer, 1024);
    12    uv_fs_read(loop, &stdin_watcher, 0, &buf, 1, -1, on_type);
    13    uv_idle_start(&idler, crunch_away);
    14    return uv_run(loop, UV_RUN_DEFAULT);
    15}

Here we initialize the idle watcher and queue it up along with the actual events we are interested in. `crunch_away` will now be called repeatedly until the user types something and presses Return. Then it will be interrupted for a brief amount as the loop deals with the input data, after which it will keep calling the idle callback again.

idle-compute/main.c

     1void crunch_away(uv_idle_t* handle) {
     2    // Compute extra-terrestrial life
     3    // fold proteins
     4    // computer another digit of PI
     5    // or similar
     6    fprintf(stderr, "Computing PI...\n");
     7    // just to avoid overwhelming your terminal emulator
     8    uv_idle_stop(handle);
     9}
    10

## Passing data to worker thread

When using `uv_queue_work` you’ll usually need to pass complex data through to the worker thread. The solution is to use a `struct` and set `uv_work_t.data` to point to it. A slight variation is to have the `uv_work_t` itself as the first member of this struct (called a baton [1]). This allows cleaning up the work request and all the data in one free call.

    1struct ftp_baton {
    2    uv_work_t req;
    3    char *host;
    4    int port;
    5    char *username;
    6    char *password;
    7}

    1ftp_baton *baton = (ftp_baton*) malloc(sizeof(ftp_baton));
    2baton->req.data = (void*) baton;
    3baton->host = strdup("my.webhost.com");
    4baton->port = 21;
    5// ...
    6
    7uv_queue_work(loop, &baton->req, ftp_session, ftp_cleanup);

Here we create the baton and queue the task.

Now the task function can extract the data it needs:

     1void ftp_session(uv_work_t *req) {
     2    ftp_baton *baton = (ftp_baton*) req->data;
     3
     4    fprintf(stderr, "Connecting to %s\n", baton->host);
     5}
     6
     7void ftp_cleanup(uv_work_t *req) {
     8    ftp_baton *baton = (ftp_baton*) req->data;
     9
    10    free(baton->host);
    11    // ...
    12    free(baton);
    13}

We then free the baton which also frees the watcher.

## External I/O with polling

Usually third-party libraries will handle their own I/O, and keep track of their sockets and other files internally. In this case it isn’t possible to use the standard stream I/O operations, but the library can still be integrated into the libuv event loop. All that is required is that the library allow you to access the underlying file descriptors and provide functions that process tasks in small increments as decided by your application. Some libraries though will not allow such access, providing only a standard blocking function which will perform the entire I/O transaction and only then return. It is unwise to use these in the event loop thread, use the Thread pool work scheduling instead. Of course, this will also mean losing granular control on the library.

The `uv_poll` section of libuv simply watches file descriptors using the operating system notification mechanism. In some sense, all the I/O operations that libuv implements itself are also backed by `uv_poll` like code. Whenever the OS notices a change of state in file descriptors being polled, libuv will invoke the associated callback.

Here we will walk through a simple download manager that will use libcurl to download files. Rather than give all control to libcurl, we’ll instead be using the libuv event loop, and use the non-blocking, async multi interface to progress with the download whenever libuv notifies of I/O readiness.

uvwget/main.c - The setup

     1#include <assert.h>
     2#include <stdio.h>
     3#include <stdlib.h>
     4#include <uv.h>
     5#include <curl/curl.h>
     6
     7uv_loop_t *loop;
     8CURLM *curl_handle;
     9uv_timer_t timeout;
    10int main(int argc, char **argv) {
    11    loop = uv_default_loop();
    12
    13    if (argc <= 1)
    14        return 0;
    15
    16    if (curl_global_init(CURL_GLOBAL_ALL)) {
    17        fprintf(stderr, "Could not init cURL\n");
    18        return 1;
    19    }
    20
    21    uv_timer_init(loop, &timeout);
    22
    23    curl_handle = curl_multi_init();
    24    curl_multi_setopt(curl_handle, CURLMOPT_SOCKETFUNCTION, handle_socket);
    25    curl_multi_setopt(curl_handle, CURLMOPT_TIMERFUNCTION, start_timeout);
    26
    27    while (argc-- > 1) {
    28        add_download(argv[argc], argc);
    29    }
    30
    31    uv_run(loop, UV_RUN_DEFAULT);
    32    curl_multi_cleanup(curl_handle);
    33    return 0;
    34}

The way each library is integrated with libuv will vary. In the case of libcurl, we can register two callbacks. The socket callback `handle_socket` is invoked whenever the state of a socket changes and we have to start polling it. `start_timeout` is called by libcurl to notify us of the next timeout interval, after which we should drive libcurl forward regardless of I/O status. This is so that libcurl can handle errors or do whatever else is required to get the download moving.

Our downloader is to be invoked as:

    $ ./uvwget [url1] [url2] ...

So we add each argument as a URL

uvwget/main.c - Adding urls

     1
     2void add_download(const char *url, int num) {
     3    char filename[50];
     4    sprintf(filename, "%d.download", num);
     5    FILE *file;
     6
     7    file = fopen(filename, "w");
     8    if (file == NULL) {
     9        fprintf(stderr, "Error opening %s\n", filename);
    10        return;
    11    }
    12
    13    CURL *handle = curl_easy_init();
    14    curl_easy_setopt(handle, CURLOPT_WRITEDATA, file);
    15    curl_easy_setopt(handle, CURLOPT_URL, url);
    16    curl_multi_add_handle(curl_handle, handle);
    17    fprintf(stderr, "Added download %s -> %s\n", url, filename);
    18}

We let libcurl directly write the data to a file, but much more is possible if you so desire.

`start_timeout` will be called immediately the first time by libcurl, so things are set in motion. This simply starts a libuv timer which drives `curl_multi_socket_action` with `CURL_SOCKET_TIMEOUT` whenever it times out. `curl_multi_socket_action` is what drives libcurl, and what we call whenever sockets change state. But before we go into that, we need to poll on sockets whenever `handle_socket` is called.

uvwget/main.c - Setting up polling

     1
     2void start_timeout(CURLM *multi, long timeout_ms, void *userp) {
     3    if (timeout_ms <= 0)
     4        timeout_ms = 1; /* 0 means directly call socket_action, but we'll do it in a bit */
     5    uv_timer_start(&timeout, on_timeout, timeout_ms, 0);
     6}
     7
     8int handle_socket(CURL *easy, curl_socket_t s, int action, void *userp, void *socketp) {
     9    curl_context_t *curl_context;
    10    if (action == CURL_POLL_IN || action == CURL_POLL_OUT) {
    11        if (socketp) {
    12            curl_context = (curl_context_t*) socketp;
    13        }
    14        else {
    15            curl_context = create_curl_context(s);
    16            curl_multi_assign(curl_handle, s, (void *) curl_context);
    17        }
    18    }
    19
    20    switch (action) {
    21        case CURL_POLL_IN:
    22            uv_poll_start(&curl_context->poll_handle, UV_READABLE, curl_perform);
    23            break;
    24        case CURL_POLL_OUT:
    25            uv_poll_start(&curl_context->poll_handle, UV_WRITABLE, curl_perform);
    26            break;
    27        case CURL_POLL_REMOVE:
    28            if (socketp) {
    29                uv_poll_stop(&((curl_context_t*)socketp)->poll_handle);
    30                destroy_curl_context((curl_context_t*) socketp);
    31                curl_multi_assign(curl_handle, s, NULL);
    32            }
    33            break;
    34        default:
    35            abort();
    36    }
    37
    38    return 0;
    39}

We are interested in the socket fd `s`, and the `action`. For every socket we create a `uv_poll_t` handle if it doesn’t exist, and associate it with the socket using `curl_multi_assign`. This way `socketp` points to it whenever the callback is invoked.

In the case that the download is done or fails, libcurl requests removal of the poll. So we stop and free the poll handle.

Depending on what events libcurl wishes to watch for, we start polling with `UV_READABLE` or `UV_WRITABLE`. Now libuv will invoke the poll callback whenever the socket is ready for reading or writing. Calling `uv_poll_start` multiple times on the same handle is acceptable, it will just update the events mask with the new value. `curl_perform` is the crux of this program.

uvwget/main.c - Driving libcurl.

     1void curl_perform(uv_poll_t *req, int status, int events) {
     2    uv_timer_stop(&timeout);
     3    int running_handles;
     4    int flags = 0;
     5    if (status < 0)                      flags = CURL_CSELECT_ERR;
     6    if (!status && events & UV_READABLE) flags |= CURL_CSELECT_IN;
     7    if (!status && events & UV_WRITABLE) flags |= CURL_CSELECT_OUT;
     8
     9    curl_context_t *context;
    10
    11    context = (curl_context_t*)req;
    12
    13    curl_multi_socket_action(curl_handle, context->sockfd, flags, &running_handles);
    14    check_multi_info();
    15}

The first thing we do is to stop the timer, since there has been some progress in the interval. Then depending on what event triggered the callback, we set the correct flags. Then we call `curl_multi_socket_action` with the socket that progressed and the flags informing about what events happened. At this point libcurl does all of its internal tasks in small increments, and will attempt to return as fast as possible, which is exactly what an evented program wants in its main thread. libcurl keeps queueing messages into its own queue about transfer progress. In our case we are only interested in transfers that are completed. So we extract these messages, and clean up handles whose transfers are done.

uvwget/main.c - Reading transfer status.

     1void check_multi_info(void) {
     2    char *done_url;
     3    CURLMsg *message;
     4    int pending;
     5
     6    while ((message = curl_multi_info_read(curl_handle, &pending))) {
     7        switch (message->msg) {
     8        case CURLMSG_DONE:
     9            curl_easy_getinfo(message->easy_handle, CURLINFO_EFFECTIVE_URL,
    10                            &done_url);
    11            printf("%s DONE\n", done_url);
    12
    13            curl_multi_remove_handle(curl_handle, message->easy_handle);
    14            curl_easy_cleanup(message->easy_handle);
    15            break;
    16
    17        default:
    18            fprintf(stderr, "CURLMSG default\n");
    19            abort();
    20        }
    21    }
    22}

## Check & Prepare watchers

TODO

## Loading libraries

libuv provides a cross platform API to dynamically load shared libraries. This can be used to implement your own plugin/extension/module system and is used by node.js to implement `require()` support for bindings. The usage is quite simple as long as your library exports the right symbols. Be careful with sanity and security checks when loading third party code, otherwise your program will behave unpredictably. This example implements a very simple plugin system which does nothing except print the name of the plugin.

Let us first look at the interface provided to plugin authors.

plugin/plugin.h

    1#ifndef UVBOOK_PLUGIN_SYSTEM
    2#define UVBOOK_PLUGIN_SYSTEM
    3
    4// Plugin authors should use this to register their plugins with mfp.
    5void mfp_register(const char *name);
    6
    7#endif

You can similarly add more functions that plugin authors can use to do useful things in your application [2]. A sample plugin using this API is:

plugin/hello.c

    1#include "plugin.h"
    2
    3void initialize() {
    4    mfp_register("Hello World!");
    5}

Our interface defines that all plugins should have an `initialize` function which will be called by the application. This plugin is compiled as a shared library and can be loaded by running our application:

    $ ./plugin libhello.dylib
    Loading libhello.dylib
    Registered plugin "Hello World!"

Note

The shared library filename will be different depending on platforms. On Linux it is `libhello.so`.

This is done by using `uv_dlopen` to first load the shared library `libhello.dylib`. Then we get access to the `initialize` function using `uv_dlsym` and invoke it.

plugin/main.c

     1#include "plugin.h"
     2
     3typedef void (*init_plugin_function)();
     4
     5void mfp_register(const char *name) {
     6    fprintf(stderr, "Registered plugin \"%s\"\n", name);
     7}
     8
     9int main(int argc, char **argv) {
    10    if (argc == 1) {
    11        fprintf(stderr, "Usage: %s [plugin1] [plugin2] ...\n", argv[0]);
    12        return 0;
    13    }
    14
    15    uv_lib_t lib;
    16    while (--argc) {
    17        fprintf(stderr, "Loading %s\n", argv[argc]);
    18        if (uv_dlopen(argv[argc], &lib)) {
    19            fprintf(stderr, "Error: %s\n", uv_dlerror(&lib));
    20            continue;
    21        }
    22
    23        init_plugin_function init_plugin;
    24        if (uv_dlsym(&lib, "initialize", (void **) &init_plugin)) {
    25            fprintf(stderr, "dlsym error: %s\n", uv_dlerror(&lib));
    26            continue;
    27        }
    28
    29        init_plugin();
    30    }
    31    return 0;
    32}

`uv_dlopen` expects a path to the shared library and sets the opaque `uv_lib_t` pointer. It returns 0 on success, -1 on error. Use `uv_dlerror` to get the error message.

`uv_dlsym` stores a pointer to the symbol in the second argument in the third argument. `init_plugin_function` is a function pointer to the sort of function we are looking for in the application’s plugins.

## TTY

Text terminals have supported basic formatting for a long time, with a pretty standardised command set. This formatting is often used by programs to improve the readability of terminal output. For example `grep --colour`. libuv provides the `uv_tty_t` abstraction (a stream) and related functions to implement the ANSI escape codes across all platforms. By this I mean that libuv converts ANSI codes to the Windows equivalent, and provides functions to get terminal information.

The first thing to do is to initialize a `uv_tty_t` with the file descriptor it reads/writes from. This is achieved with:

    int uv_tty_init(uv_loop_t*, uv_tty_t*, uv_file fd, int unused)

The `unused` parameter is now auto-detected and ignored. It previously needed to be set to use `uv_read_start()` on the stream.

It is then best to use `uv_tty_set_mode` to set the mode to normal which enables most TTY formatting, flow-control and other settings. Other modes are also available.

Remember to call `uv_tty_reset_mode` when your program exits to restore the state of the terminal. Just good manners. Another set of good manners is to be aware of redirection. If the user redirects the output of your command to a file, control sequences should not be written as they impede readability and `grep`. To check if the file descriptor is indeed a TTY, call `uv_guess_handle` with the file descriptor and compare the return value with `UV_TTY`.

Here is a simple example which prints white text on a red background:

tty/main.c

     1#include <stdio.h>
     2#include <string.h>
     3#include <unistd.h>
     4#include <uv.h>
     5
     6uv_loop_t *loop;
     7uv_tty_t tty;
     8
     9int main() {
    10    uv_write_t req;
    11    uv_buf_t buf;
    12    uv_write_t req1;
    13    uv_buf_t buf1;
    14
    15    loop = uv_default_loop();
    16    uv_tty_init(loop, &tty, STDOUT_FILENO, 0);
    17    uv_tty_set_mode(&tty, UV_TTY_MODE_NORMAL);
    18
    19    if (uv_guess_handle(1) == UV_TTY) {
    20        buf1.base = "\033[41;37m";
    21        buf1.len = strlen(buf1.base);
    22        uv_write(&req1, (uv_stream_t*) &tty, &buf1, 1, NULL);
    23    }
    24
    25    buf.base = "Hello TTY\n";
    26    buf.len = strlen(buf.base);
    27    uv_write(&req, (uv_stream_t*) &tty, &buf, 1, NULL);
    28
    29    uv_tty_reset_mode();
    30    return uv_run(loop, UV_RUN_DEFAULT);
    31}

The final TTY helper is `uv_tty_get_winsize()` which is used to get the width and height of the terminal and returns `0` on success. Here is a small program which does some animation using the function and character position escape codes.

tty-gravity/main.c

     1#include <stdio.h>
     2#include <string.h>
     3#include <unistd.h>
     4#include <uv.h>
     5
     6uv_loop_t *loop;
     7uv_tty_t tty;
     8uv_timer_t tick;
     9uv_write_t write_req;
    10int width, height;
    11int pos = 0;
    12char *message = "  Hello TTY  ";
    13
    14void update(uv_timer_t *req) {
    15    char data[500];
    16
    17    uv_buf_t buf;
    18    buf.base = data;
    19    buf.len = sprintf(data, "\033[2J\033[H\033[%dB\033[%luC\033[42;37m%s",
    20                            pos,
    21                            (unsigned long) (width-strlen(message))/2,
    22                            message);
    23    uv_write(&write_req, (uv_stream_t*) &tty, &buf, 1, NULL);
    24
    25    pos++;
    26    if (pos > height) {
    27        uv_tty_reset_mode();
    28        uv_timer_stop(&tick);
    29    }
    30}
    31
    32int main() {
    33    loop = uv_default_loop();
    34
    35    uv_tty_init(loop, &tty, STDOUT_FILENO, 0);
    36    uv_tty_set_mode(&tty, 0);
    37
    38    if (uv_tty_get_winsize(&tty, &width, &height)) {
    39        fprintf(stderr, "Could not get TTY information\n");
    40        uv_tty_reset_mode();
    41        return 1;
    42    }
    43
    44    fprintf(stderr, "Width %d, height %d\n", width, height);
    45    uv_timer_init(loop, &tick);
    46    uv_timer_start(&tick, update, 200, 200);
    47    return uv_run(loop, UV_RUN_DEFAULT);
    48}

The escape codes are:

Code | Meaning
--- | ---
2 J | Clear part of the screen, 2 is entire screen
H | Moves cursor to certain position, default top-left
n B | Moves cursor down by n lines
n C | Moves cursor right by n columns
m | Obeys string of display settings, in this case green background (40+2), white text (30+7)

As you can see this is very useful to produce nicely formatted output, or even console based arcade games if that tickles your fancy. For fancier control you can try ncurses.

Changed in version 1.23.1:: the readable parameter is now unused and ignored. The appropriate value will now be auto-detected from the kernel.

[1]

I was first introduced to the term baton in this context, in Konstantin Käfer’s excellent slides on writing node.js bindings – https://kkaefer.com/node-cpp-modules/#baton

[2]

mfp is My Fancy Plugin