# `tornado.httpclient` — Asynchronous HTTP client

Blocking and non-blocking HTTP client interfaces.

This module defines a common interface shared by two implementations, `simple_httpclient` and `curl_httpclient`. Applications may either instantiate their chosen implementation class directly or use the `AsyncHTTPClient` class from this module, which selects an implementation that can be overridden with the `AsyncHTTPClient.configure` method.

The default implementation is `simple_httpclient`, and this is expected to be suitable for most users’ needs. However, some applications may wish to switch to `curl_httpclient` for reasons such as the following:

- `curl_httpclient` has some features not found in `simple_httpclient`, including support for HTTP proxies and the ability to use a specified network interface.
- `curl_httpclient` is more likely to be compatible with sites that are not-quite-compliant with the HTTP spec, or sites that use little-exercised features of HTTP.
- `curl_httpclient` is faster.
- `curl_httpclient` was the default prior to Tornado 2.0.

Note that if you are using `curl_httpclient`, it is highly recommended that you use a recent version of `libcurl` and `pycurl`. Currently the minimum supported version of libcurl is 7.22.0, and the minimum version of pycurl is 7.18.2. It is highly recommended that your `libcurl` installation is built with asynchronous DNS resolver (threaded or c-ares), otherwise you may encounter various problems with request timeouts (for more information, see http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTCONNECTTIMEOUTMS and comments in curl_httpclient.py).

To select `curl_httpclient`, call `AsyncHTTPClient.configure` at startup:

    AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")

## HTTP client interfaces

class `tornado.httpclient.``HTTPClient`(async_client_class=None, **kwargs)

  A blocking HTTP client.

  This interface is provided to make it easier to share code between synchronous and asynchronous applications. Applications that are running an `IOLoop` must use `AsyncHTTPClient` instead.

  Typical usage looks like this:

      http_client = httpclient.HTTPClient()
      try:
          response = http_client.fetch("http://www.google.com/")
          print(response.body)
      except httpclient.HTTPError as e:
          # HTTPError is raised for non-200 responses; the response
          # can be found in e.response.
          print("Error: " + str(e))
      except Exception as e:
          # Other errors are possible, such as IOError.
          print("Error: " + str(e))
      http_client.close()

  Changed in version 5.0: Due to limitations in `asyncio`, it is no longer possible to use the synchronous `HTTPClient` while an `IOLoop` is running. Use `AsyncHTTPClient` instead.

  `close`()

    Closes the HTTPClient, freeing any resources used.

  `fetch`(request, **kwargs)

    Executes a request, returning an `HTTPResponse`.

    The request may be either a string URL or an `HTTPRequest` object. If it is a string, we construct an `HTTPRequest` using any additional kwargs: `HTTPRequest(request, **kwargs)`

    If an error occurs during the fetch, we raise an `HTTPError` unless the `raise_error` keyword argument is set to False.

class `tornado.httpclient.``AsyncHTTPClient`

  An non-blocking HTTP client.

  Example usage:

      async def f():
          http_client = AsyncHTTPClient()
          try:
              response = await http_client.fetch("http://www.google.com")
          except Exception as e:
              print("Error: %s" % e)
          else:
              print(response.body)

  The constructor for this class is magic in several respects: It actually creates an instance of an implementation-specific subclass, and instances are reused as a kind of pseudo-singleton (one per `IOLoop`). The keyword argument `force_instance=True` can be used to suppress this singleton behavior. Unless `force_instance=True` is used, no arguments should be passed to the `AsyncHTTPClient` constructor. The implementation subclass as well as arguments to its constructor can be set with the static method `configure()`

  All `AsyncHTTPClient` implementations support a `defaults` keyword argument, which can be used to set default values for `HTTPRequest` attributes. For example:

      AsyncHTTPClient.configure(
          None, defaults=dict(user_agent="MyUserAgent"))
      # or with force_instance:
      client = AsyncHTTPClient(force_instance=True,
          defaults=dict(user_agent="MyUserAgent"))

  Changed in version 5.0: The `io_loop` argument (deprecated since version 4.1) has been removed.

  `close`()

    Destroys this HTTP client, freeing any file descriptors used.

    This method is not needed in normal use due to the way that `AsyncHTTPClient` objects are transparently reused. `close()` is generally only necessary when either the `IOLoop` is also being closed, or the `force_instance=True` argument was used when creating the `AsyncHTTPClient`.

    No other methods may be called on the `AsyncHTTPClient` after `close()`.

  `fetch`(request, callback=None, raise_error=True, **kwargs)

    Executes a request, asynchronously returning an `HTTPResponse`.

    The request may be either a string URL or an `HTTPRequest` object. If it is a string, we construct an `HTTPRequest` using any additional kwargs: `HTTPRequest(request, **kwargs)`

    This method returns a `Future` whose result is an `HTTPResponse`. By default, the `Future` will raise an `HTTPError` if the request returned a non-200 response code (other errors may also be raised if the server could not be contacted). Instead, if `raise_error` is set to False, the response will always be returned regardless of the response code.

    If a `callback` is given, it will be invoked with the `HTTPResponse`. In the callback interface, `HTTPError` is not automatically raised. Instead, you must check the response’s `error` attribute or call its `rethrow` method.

    Deprecated since version 5.1: The `callback` argument is deprecated and will be removed in 6.0. Use the returned `Future` instead.

    The `raise_error=False` argument currently suppresses all errors, encapsulating them in `HTTPResponse` objects with a 599 response code. This will change in Tornado 6.0: `raise_error=False` will only affect the `HTTPError` raised when a non-200 response code is used.

  classmethod `configure`(impl, **kwargs)

    Configures the `AsyncHTTPClient` subclass to use.

    `AsyncHTTPClient()` actually creates an instance of a subclass. This method may be called with either a class object or the fully-qualified name of such a class (or `None` to use the default, `SimpleAsyncHTTPClient`)

    If additional keyword arguments are given, they will be passed to the constructor of each subclass instance created. The keyword argument `max_clients` determines the maximum number of simultaneous `fetch()` operations that can execute in parallel on each `IOLoop`. Additional arguments may be supported depending on the implementation class in use.

    Example:

        AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")

## Request objects

class `tornado.httpclient.``HTTPRequest`(url, method='GET', headers=None, body=None, auth_username=None, auth_password=None, auth_mode=None, connect_timeout=None, request_timeout=None, if_modified_since=None, follow_redirects=None, max_redirects=None, user_agent=None, use_gzip=None, network_interface=None, streaming_callback=None, header_callback=None, prepare_curl_callback=None, proxy_host=None, proxy_port=None, proxy_username=None, proxy_password=None, proxy_auth_mode=None, allow_nonstandard_methods=None, validate_cert=None, ca_certs=None, allow_ipv6=None, client_key=None, client_cert=None, body_producer=None, expect_100_continue=False, decompress_response=None, ssl_options=None)

  HTTP client request object.

  All parameters except `url` are optional.

  Parameters:

  - url (str) – URL to fetch
  - method (str) – HTTP method, e.g. “GET” or “POST”
  - headers (`HTTPHeaders` or `dict`) – Additional HTTP headers to pass on the request
  - body – HTTP request body as a string (byte or unicode; if unicode the utf-8 encoding will be used)
  - body_producer – Callable used for lazy/asynchronous request bodies. It is called with one argument, a `write` function, and should return a `Future`. It should call the write function with new data as it becomes available. The write function returns a `Future` which can be used for flow control. Only one of `body` and `body_producer` may be specified. `body_producer` is not supported on `curl_httpclient`. When using `body_producer` it is recommended to pass a `Content-Length` in the headers as otherwise chunked encoding will be used, and many servers do not support chunked encoding on requests. New in Tornado 4.0
  - auth_username (str) – Username for HTTP authentication
  - auth_password (str) – Password for HTTP authentication
  - auth_mode (str) – Authentication mode; default is “basic”. Allowed values are implementation-defined; `curl_httpclient` supports “basic” and “digest”; `simple_httpclient` only supports “basic”
  - connect_timeout (float) – Timeout for initial connection in seconds, default 20 seconds
  - request_timeout (float) – Timeout for entire request in seconds, default 20 seconds
  - if_modified_since (`datetime` or `float`) – Timestamp for `If-Modified-Since` header
  - follow_redirects (bool) – Should redirects be followed automatically or return the 3xx response? Default True.
  - max_redirects (int) – Limit for `follow_redirects`, default 5.
  - user_agent (str) – String to send as `User-Agent` header
  - decompress_response (bool) – Request a compressed response from the server and decompress it after downloading. Default is True. New in Tornado 4.0.
  - use_gzip (bool) – Deprecated alias for `decompress_response` since Tornado 4.0.
  - network_interface (str) – Network interface to use for request. `curl_httpclient` only; see note below.
  - streaming_callback (collections.abc.Callable) – If set, `streaming_callback` will be run with each chunk of data as it is received, and `HTTPResponse.body` and `HTTPResponse.buffer` will be empty in the final response.
  - header_callback (collections.abc.Callable) – If set, `header_callback` will be run with each header line as it is received (including the first line, e.g. `HTTP/1.0 200 OK\r\n`, and a final line containing only `\r\n`. All lines include the trailing newline characters). `HTTPResponse.headers` will be empty in the final response. This is most useful in conjunction with `streaming_callback`, because it’s the only way to get access to header data while the request is in progress.
  - prepare_curl_callback (collections.abc.Callable) – If set, will be called with a `pycurl.Curl` object to allow the application to make additional `setopt` calls.
  - proxy_host (str) – HTTP proxy hostname. To use proxies, `proxy_host` and `proxy_port` must be set; `proxy_username`, `proxy_pass` and `proxy_auth_mode` are optional. Proxies are currently only supported with `curl_httpclient`.
  - proxy_port (int) – HTTP proxy port
  - proxy_username (str) – HTTP proxy username
  - proxy_password (str) – HTTP proxy password
  - proxy_auth_mode (str) – HTTP proxy Authentication mode; default is “basic”. supports “basic” and “digest”
  - allow_nonstandard_methods (bool) – Allow unknown values for `method` argument? Default is False.
  - validate_cert (bool) – For HTTPS requests, validate the server’s certificate? Default is True.
  - ca_certs (str) – filename of CA certificates in PEM format, or None to use defaults. See note below when used with `curl_httpclient`.
  - client_key (str) – Filename for client SSL key, if any. See note below when used with `curl_httpclient`.
  - client_cert (str) – Filename for client SSL certificate, if any. See note below when used with `curl_httpclient`.
  - ssl_options (ssl.SSLContext) – `ssl.SSLContext` object for use in `simple_httpclient` (unsupported by `curl_httpclient`). Overrides `validate_cert`, `ca_certs`, `client_key`, and `client_cert`.
  - allow_ipv6 (bool) – Use IPv6 when available? Default is true.
  - expect_100_continue (bool) – If true, send the `Expect: 100-continue` header and wait for a continue response before sending the request body. Only supported with simple_httpclient.

  Note

  When using `curl_httpclient` certain options may be inherited by subsequent fetches because `pycurl` does not allow them to be cleanly reset. This applies to the `ca_certs`, `client_key`, `client_cert`, and `network_interface` arguments. If you use these options, you should pass them on every request (you don’t have to always use the same values, but it’s not possible to mix requests that specify these options with ones that use the defaults).

  New in version 3.1: The `auth_mode` argument.

  New in version 4.0: The `body_producer` and `expect_100_continue` arguments.

  New in version 4.2: The `ssl_options` argument.

  New in version 4.5: The `proxy_auth_mode` argument.

## Response objects

class `tornado.httpclient.``HTTPResponse`(request, code, headers=None, buffer=None, effective_url=None, error=None, request_time=None, time_info=None, reason=None, start_time=None)

  HTTP Response object.

  Attributes:

  - request: HTTPRequest object
  - code: numeric HTTP status code, e.g. 200 or 404
  - reason: human-readable reason phrase describing the status code
  - headers: `tornado.httputil.HTTPHeaders` object
  - effective_url: final location of the resource after following any redirects
  - buffer: `cStringIO` object for response body
  - body: response body as bytes (created on demand from `self.buffer`)
  - error: Exception object, if any
  - request_time: seconds from request start to finish. Includes all network operations from DNS resolution to receiving the last byte of data. Does not include time spent in the queue (due to the `max_clients` option). If redirects were followed, only includes the final request.
  - start_time: Time at which the HTTP operation started, based on `time.time` (not the monotonic clock used by `IOLoop.time`). May be `None` if the request timed out while in the queue.
  - time_info: dictionary of diagnostic timing information from the request. Available data are subject to change, but currently uses timings available from http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html, plus `queue`, which is the delay (if any) introduced by waiting for a slot under `AsyncHTTPClient`’s `max_clients` setting.

  New in version 5.1: Added the `start_time` attribute.

  Changed in version 5.1: The `request_time` attribute previously included time spent in the queue for `simple_httpclient`, but not in `curl_httpclient`. Now queueing time is excluded in both implementations. `request_time` is now more accurate for `curl_httpclient` because it uses a monotonic clock when available.

  `rethrow`()

    If there was an error on the request, raise an `HTTPError`.

## Exceptions

exception `tornado.httpclient.``HTTPClientError`(code, message=None, response=None)

  Exception thrown for an unsuccessful HTTP request.

  Attributes:

  - `code` - HTTP error integer error code, e.g. 404. Error code 599 is used when no HTTP response was received, e.g. for a timeout.
  - `response` - `HTTPResponse` object, if any.

  Note that if `follow_redirects` is False, redirects become HTTPErrors, and you can look at `error.response.headers['Location']` to see the destination of the redirect.

  Changed in version 5.1: Renamed from `HTTPError` to `HTTPClientError` to avoid collisions with `tornado.web.HTTPError`. The name `tornado.httpclient.HTTPError` remains as an alias.

exception `tornado.httpclient.``HTTPError`

  Alias for `HTTPClientError`.

## Command-line interface

This module provides a simple command-line interface to fetch a url using Tornado’s HTTP client. Example usage:

    # Fetch the url and print its body
    python -m tornado.httpclient http://www.google.com

    # Just print the headers
    python -m tornado.httpclient --print_headers --print_body=false http://www.google.com

## Implementations

exception `tornado.simple_httpclient.``HTTPTimeoutError`(message)

  Error raised by SimpleAsyncHTTPClient on timeout.

  For historical reasons, this is a subclass of `HTTPClientError` which simulates a response code of 599.

  New in version 5.1.

exception `tornado.simple_httpclient.``HTTPStreamClosedError`(message)

  Error raised by SimpleAsyncHTTPClient when the underlying stream is closed.

  When a more specific exception is available (such as `ConnectionResetError`), it may be raised instead of this one.

  For historical reasons, this is a subclass of `HTTPClientError` which simulates a response code of 599.

  New in version 5.1.

class `tornado.simple_httpclient.``SimpleAsyncHTTPClient`

  Non-blocking HTTP client with no external dependencies.

  This class implements an HTTP 1.1 client on top of Tornado’s IOStreams. Some features found in the curl-based AsyncHTTPClient are not yet supported. In particular, proxies are not supported, connections are not reused, and callers cannot select the network interface to be used.

  `initialize`(max_clients=10, hostname_mapping=None, max_buffer_size=104857600, resolver=None, defaults=None, max_header_size=None, max_body_size=None)

    Creates a AsyncHTTPClient.

    Only a single AsyncHTTPClient instance exists per IOLoop in order to provide limitations on the number of pending connections. `force_instance=True` may be used to suppress this behavior.

    Note that because of this implicit reuse, unless `force_instance` is used, only the first call to the constructor actually uses its arguments. It is recommended to use the `configure` method instead of the constructor to ensure that arguments take effect.

    `max_clients` is the number of concurrent requests that can be in progress; when this limit is reached additional requests will be queued. Note that time spent waiting in this queue still counts against the `request_timeout`.

    `hostname_mapping` is a dictionary mapping hostnames to IP addresses. It can be used to make local DNS changes when modifying system-wide settings like `/etc/hosts` is not possible or desirable (e.g. in unittests).

    `max_buffer_size` (default 100MB) is the number of bytes that can be read into memory at once. `max_body_size` (defaults to `max_buffer_size`) is the largest response body that the client will accept. Without a `streaming_callback`, the smaller of these two limits applies; with a `streaming_callback` only `max_body_size` does.

    Changed in version 4.2: Added the `max_body_size` argument.

class `tornado.curl_httpclient.``CurlAsyncHTTPClient`(max_clients=10, defaults=None)

  `libcurl`-based HTTP client.

## Example Code

- A simple webspider shows how to fetch URLs concurrently.
- The file uploader demo uses either HTTP POST or HTTP PUT to upload files to a server.