Hide navigation sidebar
Hide table of contents sidebar
Skip to content
Back to top
Toggle table of contents sidebar

# `gevent.fileobject` – Wrappers to make file-like objects cooperative

Wrappers to make file-like objects cooperative.

class FileObject(fobj, mode='r', buffering=-1, closefd=True, encoding=None, errors=None, newline=None)

  The main entry point to the file-like gevent-compatible behaviour. It will be defined to be the best available implementation.

  All the parameters are as for `io.open()`.

  Parameters:

    fobj –

    Usually a file descriptor of a socket. Can also be another object with a `fileno()` method, or an object that can be passed to `io.open()` (e.g., a file system path). If the object is not a socket, the results will vary based on the platform and the type of object being opened.

    All supported versions of Python allow `os.PathLike` objects.

  Changed in version 1.5: Accept str and `PathLike` objects for fobj on all versions of Python.

  Changed in version 1.5: Add encoding, errors and newline arguments.

  Changed in version 1.5: Accept closefd and buffering instead of close and bufsize arguments. The latter remain for backwards compatibility.

There are two main implementations of `FileObject`. On all systems, there is `FileObjectThread` which uses the built-in native threadpool to avoid blocking the entire interpreter. On UNIX systems (those that support the `fcntl` module), there is also `FileObjectPosix` which uses native non-blocking semantics.

A third class, `FileObjectBlock`, is simply a wrapper that executes everything synchronously (and so is not gevent-compatible). It is provided for testing and debugging purposes.

All classes have the same signature; some may accept extra keyword arguments.

## Configuration

You may change the default value for `FileObject` using the `GEVENT_FILE` environment variable. Set it to `posix`, `thread`, or `block` to choose from `FileObjectPosix`, `FileObjectThread` and `FileObjectBlock`, respectively. You may also set it to the fully qualified class name of another object that implements the file interface to use one of your own objects.

Note

The environment variable must be set at the time this module is first imported.

## Classes

FileObject

  alias of `FileObjectPosix`

class FileObjectBlock

  Bases: `FileObjectBase`

  A simple synchronous wrapper around a file object.

  Adds no concurrency or gevent compatibility.

class FileObjectPosix

  Bases: `FileObjectBase`

  A file-like object that operates on non-blocking files but provides a synchronous, cooperative interface.

  Caution

  This object is only effective wrapping files that can be used meaningfully with `select.select()` such as sockets and pipes.

  In general, on most platforms, operations on regular files (e.g., `open('a_file.txt')`) are considered non-blocking already, even though they can take some time to complete as data is copied to the kernel and flushed to disk: this time is relatively bounded compared to sockets or pipes, though. A `read()` or `write()` call on such a file will still effectively block for some small period of time. Therefore, wrapping this class around a regular file is unlikely to make IO gevent-friendly: reading or writing large amounts of data could still block the event loop.

  If you’ll be working with regular files and doing IO in large chunks, you may consider using `FileObjectThread` or `tp_read()` and `tp_write()` to bypass this concern.

  Tip

  Although this object provides a `fileno()` method and so can itself be passed to `fcntl.fcntl()`, setting the `os.O_NONBLOCK` flag will have no effect (reads will still block the greenlet, although other greenlets can run). However, removing that flag will cause this object to no longer be cooperative (other greenlets will no longer run).

  You can use the internal `fileio` attribute of this object (a `io.RawIOBase`) to perform non-blocking byte reads. Note, however, that once you begin directly using this attribute, the results from using methods of this object are undefined, especially in text mode. (See issue #222.)

  Changed in version 1.1: Now uses the `io` package internally. Under Python 2, previously used the undocumented class `socket._fileobject`. This provides better file-like semantics (and portability to Python 3).

  Changed in version 1.2a1: Document the `fileio` attribute for non-blocking reads.

  Changed in version 1.2a1: A bufsize of 0 in write mode is no longer forced to be 1. Instead, the underlying buffer is flushed after every write operation to simulate a bufsize of 0. In gevent 1.0, a bufsize of 0 was flushed when a newline was written, while in gevent 1.1 it was flushed when more than one byte was written. Note that this may have performance impacts.

  Changed in version 1.3a1: On Python 2, enabling universal newlines no longer forces unicode IO.

  Changed in version 1.5: The default value for mode was changed from `rb` to `r`. This is consistent with `open()`, `io.open()`, and `FileObjectThread`, which is the default `FileObject` on some platforms.

  Changed in version 1.5: Stop forcing buffering. Previously, given a `buffering=0` argument, buffering would be set to 1, and `buffering=1` would be forced to the default buffer size. This was a workaround for a long-standing concurrency issue. Now the buffering argument is interpreted as intended.

class FileObjectThread

  Bases: `FileObjectBase`

  A file-like object wrapping another file-like object, performing all blocking operations on that object in a background thread.

  Caution

  Attempting to change the threadpool or lock of an existing FileObjectThread has undefined consequences.

  Changed in version 1.1b1: The file object is closed using the threadpool. Note that whether or not this action is synchronous or asynchronous is not documented.

  Parameters:

    - lock (bool) – If True (the default) then all operations will be performed one-by-one. Note that this does not guarantee that, if using this file object from multiple threads/greenlets, operations will be performed in any particular order, only that no two operations will be attempted at the same time. You can also pass your own `gevent.lock.Semaphore` to synchronize file operations with an external resource.

    - closefd (bool) – If True (the default) then when this object is closed, the underlying object is closed as well. If fobj is a path, then closefd must be True.