| libtorrent

Author: | Arvid Norberg, arvid@libtorrent.org
Version: | 1.1.5

home

# Custom Storage

Table of contents

- Custom Storage
  - disk_buffer_holder
  - file_pool
  - storage_interface
  - default_storage
  - enum move_flags_t

libtorrent provides a customization point for storage of data. By default, (`default_storage`) downloaded files are saved to disk according with the general conventions of bittorrent clients, mimicing the original file layout when the torrent was created. The libtorrent user may define a custom storage to store piece data in a different way.

A custom storage implementation must derive from and implement the storage_interface. You must also provide a function that constructs the custom storage object and provide this function to the add_torrent() call via add_torrent_params. Either passed in to the constructor or by setting the add_torrent_params::storage field.

This is an example storage implementation that stores all pieces in a `std::map`, i.e. in RAM. It's not necessarily very useful in practice, but illustrates the basics of implementing a custom storage.

    struct temp_storage : storage_interface
    {
            temp_storage(file_storage const& fs) : m_files(fs) {}
            virtual bool initialize(storage_error& se) { return false; }
            virtual bool has_any_file() { return false; }
            virtual int read(char* buf, int piece, int offset, int size)
            {
                    std::map<int, std::vector<char> >::const_iterator i = m_file_data.find(piece);
                    if (i == m_file_data.end()) return 0;
                    int available = i->second.size() - offset;
                    if (available <= 0) return 0;
                    if (available > size) available = size;
                    memcpy(buf, &i->second[offset], available);
                    return available;
            }
            virtual int write(const char* buf, int piece, int offset, int size)
            {
                    std::vector<char>& data = m_file_data[piece];
                    if (data.size() < offset + size) data.resize(offset + size);
                    std::memcpy(&data[offset], buf, size);
                    return size;
            }
            virtual bool rename_file(int file, std::string const& new_name)
            { assert(false); return false; }
            virtual bool move_storage(std::string const& save_path) { return false; }
            virtual bool verify_resume_data(bdecode_node const& rd
                    , std::vector<std::string> const* links
                    , storage_error& error) { return false; }
            virtual bool write_resume_data(entry& rd) const { return false; }
            virtual boost::int64_t physical_offset(int piece, int offset)
            { return piece * m_files.piece_length() + offset; };
            virtual sha1_hash hash_for_slot(int piece, partial_hash& ph, int piece_size)
            {
                    int left = piece_size - ph.offset;
                    assert(left >= 0);
                    if (left > 0)
                    {
                            std::vector<char>& data = m_file_data[piece];
                            // if there are padding files, those blocks will be considered
                            // completed even though they haven't been written to the storage.
                            // in this case, just extend the piece buffer to its full size
                            // and fill it with zeros.
                            if (data.size() < piece_size) data.resize(piece_size, 0);
                            ph.h.update(&data[ph.offset], left);
                    }
                    return ph.h.final();
            }
            virtual bool release_files() { return false; }
            virtual bool delete_files() { return false; }

            std::map<int, std::vector<char> > m_file_data;
            file_storage m_files;
    };

    storage_interface* temp_storage_constructor(storage_params const& params)
    {
            return new temp_storage(*params.files);
    }

## disk_buffer_holder

Declared in "libtorrent/disk_buffer_holder.hpp"

The disk buffer holder acts like a `scoped_ptr` that frees a disk buffer when it's destructed, unless it's released. `release` returns the disk buffer and transfers ownership and responsibility to free it to the caller.

A disk buffer is freed by passing it to `session_impl::free_disk_buffer()`.

`get()` returns the pointer without transferring responsibility. If this buffer has been released, `buffer()` will return 0.

    struct disk_buffer_holder
    {
       disk_buffer_holder (buffer_allocator_interface& alloc, disk_io_job const& j);
       ~disk_buffer_holder ();
       char* release ();
       char* get () const;
       void reset (char* buf = 0);
       void reset (disk_io_job const& j);
       void swap (disk_buffer_holder& h);
       block_cache_reference ref () const;
    };

### disk_buffer_holder()

    disk_buffer_holder (buffer_allocator_interface& alloc, disk_io_job const& j);

construct a buffer holder that will free the held buffer using a disk buffer pool directly (there's only one disk_buffer_pool per session)

### ~disk_buffer_holder()

    ~disk_buffer_holder ();

frees any unreleased disk buffer held by this object

### release()

    char* release ();

return the held disk buffer and clear it from the holder. The responsibility to free it is passed on to the caller

### get()

    char* get () const;

return a pointer to the held buffer

### reset()

    void reset (char* buf = 0);
    void reset (disk_io_job const& j);

set the holder object to hold the specified buffer (or NULL by default). If it's already holding a disk buffer, it will first be freed.

### swap()

    void swap (disk_buffer_holder& h);

swap pointers of two disk buffer holders.

## file_pool

Declared in "libtorrent/file_pool.hpp"

this is an internal cache of open file handles. It's primarily used by storage_interface implementations. It provides semi weak guarantees of not opening more file handles than specified. Given multiple threads, each with the ability to lock a file handle (via smart pointer), there may be windows where more file handles are open.

    struct file_pool : boost::noncopyable
    {
       ~file_pool ();
       file_pool (int size = 40);
       file_handle open_file (void* st, std::string const& p
          , int file_index, file_storage const& fs, int m, error_code& ec);
       void release (void* st = NULL);
       void release (void* st, int file_index);
       void resize (int size);
       int size_limit () const;
       void close_oldest ();
    };

### ~file_pool() file_pool()

    ~file_pool ();
    file_pool (int size = 40);

`size` specifies the number of allowed files handles to hold open at any given time.

### open_file()

    file_handle open_file (void* st, std::string const& p
          , int file_index, file_storage const& fs, int m, error_code& ec);

return an open file handle to file at `file_index` in the file_storage `fs` opened at save path `p`. `m` is the file open mode (see file::open_mode_t).

### release()

    void release (void* st = NULL);
    void release (void* st, int file_index);

release all files belonging to the specified storage_interface (`st`) the overload that takes `file_index` releases only the file with that index in storage `st`.

### resize()

    void resize (int size);

update the allowed number of open file handles to `size`.

### size_limit()

    int size_limit () const;

returns the current limit of number of allowed open file handles held by the file_pool.

### close_oldest()

    void close_oldest ();

close the file that was opened least recently (i.e. not accessed least recently). The purpose is to make the OS (really just windows) clear and flush its disk cache associated with this file. We don't want any file to stay open for too long, allowing the disk cache to accrue.

## storage_interface

Declared in "libtorrent/storage.hpp"

The storage interface is a pure virtual class that can be implemented to customize how and where data for a torrent is stored. The default storage implementation uses regular files in the filesystem, mapping the files in the torrent in the way one would assume a torrent is saved to disk. Implementing your own storage interface makes it possible to store all data in RAM, or in some optimized order on disk (the order the pieces are received for instance), or saving multi file torrents in a single file in order to be able to take advantage of optimized disk-I/O.

It is also possible to write a thin class that uses the default storage but modifies some particular behavior, for instance encrypting the data before it's written to disk, and decrypting it when it's read again.

The storage interface is based on pieces. Avery read and write operation happens in the piece-space. Each piece fits 'piece_size' number of bytes. All access is done by writing and reading whole or partial pieces.

libtorrent comes with two built-in storage implementations; `default_storage` and `disabled_storage`. Their constructor functions are called default_storage_constructor() and `disabled_storage_constructor` respectively. The disabled storage does just what it sounds like. It throws away data that's written, and it reads garbage. It's useful mostly for benchmarking and profiling purpose.

    struct storage_interface
    {
       virtual void initialize (storage_error& ec) = 0;
       virtual int writev (file::iovec_t const* bufs, int num_bufs
          , int piece, int offset, int flags, storage_error& ec) = 0;
       virtual int readv (file::iovec_t const* bufs, int num_bufs
          , int piece, int offset, int flags, storage_error& ec) = 0;
       virtual bool has_any_file (storage_error& ec) = 0;
       virtual void set_file_priority (std::vector<boost::uint8_t> const& prio
          , storage_error& ec) = 0;
       virtual int move_storage (std::string const& save_path, int flags
          , storage_error& ec) = 0;
       virtual bool verify_resume_data (bdecode_node const& rd
          , std::vector<std::string> const* links
          , storage_error& ec) = 0;
       virtual void write_resume_data (entry& rd, storage_error& ec) const = 0;
       virtual void release_files (storage_error& ec) = 0;
       virtual void rename_file (int index, std::string const& new_filename
          , storage_error& ec) = 0;
       virtual void delete_files (int options, storage_error& ec) = 0;
       virtual bool tick ();
       aux::session_settings const& settings () const;

       aux::session_settings* m_settings;
    };

### initialize()

    virtual void initialize (storage_error& ec) = 0;

This function is called when the storage is to be initialized. The default storage will create directories and empty files at this point. If `allocate_files` is true, it will also `ftruncate` all files to their target size.

If an error occurs, `storage_error` should be set to reflect it.

### writev() readv()

    virtual int writev (file::iovec_t const* bufs, int num_bufs
          , int piece, int offset, int flags, storage_error& ec) = 0;
    virtual int readv (file::iovec_t const* bufs, int num_bufs
          , int piece, int offset, int flags, storage_error& ec) = 0;

These functions should read and write the data in or to the given `piece` at the given `offset`. It should read or write `num_bufs` buffers sequentially, where the size of each buffer is specified in the buffer array `bufs`. The file::iovec_t type has the following members:

    struct iovec_t { void* iov_base; size_t iov_len; };

These functions may be called simultaneously from multiple threads. Make sure they are thread safe. The `file` in libtorrent is thread safe when it can fall back to `pread`, `preadv` or the windows equivalents. On targets where read operations cannot be thread safe (i.e one has to seek first and then read), only one disk thread is used.

Every buffer in `bufs` can be assumed to be page aligned and be of a page aligned size, except for the last buffer of the torrent. The allocated buffer can be assumed to fit a fully page aligned number of bytes though. This is useful when reading and writing the last piece of a file in unbuffered mode.

The `offset` is aligned to 16 kiB boundaries most of the time, but there are rare exceptions when it's not. Specifically if the read cache is disabled/or full and a peer requests unaligned data. Most clients request aligned data.

The number of bytes read or written should be returned, or -1 on error. If there's an error, the `storage_error` must be filled out to represent the error that occurred.

### has_any_file()

    virtual bool has_any_file (storage_error& ec) = 0;

This function is called when first checking (or re-checking) the storage for a torrent. It should return true if any of the files that is used in this storage exists on disk. If so, the storage will be checked for existing pieces before starting the download.

If an error occurs, `storage_error` should be set to reflect it.

### set_file_priority()

    virtual void set_file_priority (std::vector<boost::uint8_t> const& prio
          , storage_error& ec) = 0;

change the priorities of files. This is a fenced job and is guaranteed to be the only running function on this storage when called

### move_storage()

    virtual int move_storage (std::string const& save_path, int flags
          , storage_error& ec) = 0;

This function should move all the files belonging to the storage to the new save_path. The default storage moves the single file or the directory of the torrent.

Before moving the files, any open file handles may have to be closed, like `release_files()`.

If an error occurs, `storage_error` should be set to reflect it.

returns one of: | no_error = 0 | fatal_disk_error = -1 | need_full_check = -2 | file_exist = -4

### verify_resume_data()

    virtual bool verify_resume_data (bdecode_node const& rd
          , std::vector<std::string> const* links
          , storage_error& ec) = 0;

This function should verify the resume data `rd` with the files on disk. If the resume data seems to be up-to-date, return true. If not, set `error` to a description of what mismatched and return false.

The default storage may compare file sizes and time stamps of the files.

If an error occurs, `storage_error` should be set to reflect it.

This function should verify the resume data `rd` with the files on disk. If the resume data seems to be up-to-date, return true. If not, set `error` to a description of what mismatched and return false.

If the `links` pointer is non-null, it has the same number of elements as there are files. Each element is either empty or contains the absolute path to a file identical to the corresponding file in this torrent. The storage must create hard links (or copy) those files. If any file does not exist or is inaccessible, the disk job must fail.

### write_resume_data()

    virtual void write_resume_data (entry& rd, storage_error& ec) const = 0;

This function should fill in resume data, the current state of the storage, in `rd`. The default storage adds file timestamps and sizes.

Returning `true` indicates an error occurred.

If an error occurs, `storage_error` should be set to reflect it.

### release_files()

    virtual void release_files (storage_error& ec) = 0;

This function should release all the file handles that it keeps open to files belonging to this storage. The default implementation just calls file_pool::release_files().

If an error occurs, `storage_error` should be set to reflect it.

### rename_file()

    virtual void rename_file (int index, std::string const& new_filename
          , storage_error& ec) = 0;

Rename the file with index `file` to name `new_name`.

If an error occurs, `storage_error` should be set to reflect it.

### delete_files()

    virtual void delete_files (int options, storage_error& ec) = 0;

This function should delete some or all of the storage for this torrent. The `options` parameter specifies whether to delete all files or just the partfile. `options` are set to the same value as the options passed to session::remove_torrent().

If an error occurs, `storage_error` should be set to reflect it.

The `disk_buffer_pool` is used to allocate and free disk buffers. It has the following members:

    struct disk_buffer_pool : boost::noncopyable
    {
            char* allocate_buffer(char const* category);
            void free_buffer(char* buf);

            char* allocate_buffers(int blocks, char const* category);
            void free_buffers(char* buf, int blocks);

            int block_size() const { return m_block_size; }

            void release_memory();
    };

### tick()

    virtual bool tick ();

called periodically (useful for deferred flushing). When returning false, it means no more ticks are necessary. Any disk job submitted will re-enable ticking. The default will always turn ticking back off again.

### settings()

    aux::session_settings const& settings () const;

access global session_settings

m_settings
  initialized in disk_io_thread::perform_async_job

## default_storage

Declared in "libtorrent/storage.hpp"

The default implementation of storage_interface. Behaves as a normal bittorrent client. It is possible to derive from this class in order to override some of its behavior, when implementing a custom storage.

    class default_storage : public storage_interface, boost::noncopyable
    {
       default_storage (storage_params const& params);
       virtual int move_storage (std::string const& save_path, int flags
          , storage_error& ec) override;
       virtual bool verify_resume_data (bdecode_node const& rd
          , std::vector<std::string> const* links
          , storage_error& error) override;
       virtual void initialize (storage_error& ec) override;
       virtual bool tick () override;
       virtual void delete_files (int options, storage_error& ec) override;
       virtual void rename_file (int index, std::string const& new_filename
          , storage_error& ec) override;
       virtual void set_file_priority (std::vector<boost::uint8_t> const& prio
          , storage_error& ec) override;
       virtual void write_resume_data (entry& rd, storage_error& ec) const override;
       virtual void release_files (storage_error& ec) override;
       virtual bool has_any_file (storage_error& ec) override;
       int readv (file::iovec_t const* bufs, int num_bufs
          , int piece, int offset, int flags, storage_error& ec) override;
       int writev (file::iovec_t const* bufs, int num_bufs
          , int piece, int offset, int flags, storage_error& ec) override;
       file_storage const& files () const;
       static void disk_write_access_log (bool enable);
       static bool disk_write_access_log ();
    };

### default_storage()

    default_storage (storage_params const& params);

constructs the default_storage based on the give file_storage (fs). `mapped` is an optional argument (it may be NULL). If non-NULL it represents the file mapping that have been made to the torrent before adding it. That's where files are supposed to be saved and looked for on disk. `save_path` is the root save folder for this torrent. `file_pool` is the cache of file handles that the storage will use. All files it opens will ask the file_pool to open them. `file_prio` is a vector indicating the priority of files on startup. It may be an empty vector. Any file whose index is not represented by the vector (because the vector is too short) are assumed to have priority 1. this is used to treat files with priority 0 slightly differently.

### files()

    file_storage const& files () const;

if the files in this storage are mapped, returns the mapped file_storage, otherwise returns the original file_storage object.

## enum move_flags_t

Declared in "libtorrent/storage.hpp"

name | value | description
--- | --- | ---
always_replace_files | 0 | replace any files in the destination when copying or moving the storage
fail_if_exist | 1 | if any files that we want to copy exist in the destination exist, fail the whole operation and don't perform any copy or move. There is an inherent race condition in this mode. The files are checked for existence before the operation starts. In between the check and performing the copy, the destination files may be created, in which case they are replaced.
dont_replace | 2 | if any file exist in the target, take those files instead of the ones we may have in the source.