# Supporting additional requests

To support additional requests, your framework must provide the request download implementation of `HttpClient`.

## Providing the Downloader

On its own, `HttpClient` doesn’t do anything. It doesn’t know how to execute the request on its own. Thus, for frameworks or projects wanting to use additional requests in Page Objects, they need to set the implementation on how to execute an `HttpRequest`.

For more info on this, kindly read the API Specifications for `HttpClient`.

In any case, frameworks that wish to support web-poet could provide the HTTP downloader implementation in two ways:

### 1. Context Variable

`contextvars` is natively supported in `asyncio` in order to set and access context-aware values. This means that the framework using web-poet can assign the request downloader implementation using the `contextvars` instance named `web_poet.request_downloader_var`.

This can be set using:

    import attrs
    import web_poet
    from web_poet import validates_input

    async def request_implementation(
        req: web_poet.HttpRequest,
    ) -> web_poet.HttpResponse: ...

    def create_http_client():
        return web_poet.HttpClient()

    @attrs.define
    class SomePage(web_poet.WebPage):
        http: web_poet.HttpClient

        @validates_input
        async def to_item(self): ...

    # Once this is set, the ``request_implementation`` becomes available to
    # all instances of HttpClient, unless HttpClient is created with
    # the ``request_downloader`` argument (see the #2 Dependency Injection
    # example below).
    web_poet.request_downloader_var.set(request_implementation)

    # Assume that it's constructed with the necessary arguments taken somewhere.
    response = web_poet.HttpResponse(...)

    page = SomePage(response=response, http=create_http_client())
    item = await page.to_item()

When the `web_poet.request_downloader_var` contextvar is set, `HttpClient` instances use it by default.

Warning

If no value for `web_poet.request_downloader_var` is set, then `RequestDownloaderVarError` is raised. However, no exception is raised if option 2 below is used.

### 2. Dependency Injection

The framework using web-poet may be using libraries that don’t have a full support to `contextvars` (e.g. Twisted). With that, an alternative approach would be to supply the request downloader implementation when creating an `HttpClient` instance:

    import attrs
    import web_poet
    from web_poet import validates_input

    async def request_implementation(
        req: web_poet.HttpRequest,
    ) -> web_poet.HttpResponse: ...

    def create_http_client():
        return web_poet.HttpClient(request_downloader=request_implementation)

    @attrs.define
    class SomePage(web_poet.WebPage):
        http: web_poet.HttpClient

        @validates_input
        async def to_item(self): ...

    # Assume that it's constructed with the necessary arguments taken somewhere.
    response = web_poet.HttpResponse(...)

    page = SomePage(response=response, http=create_http_client())
    item = await page.to_item()

From the code sample above, we can see that every time an `HttpClient` instance is created for Page Objects needing it, the framework must create `HttpClient` with a framework-specific request downloader implementation, using the `request_downloader` argument.

## Downloader Behavior

The request downloader MUST accept an instance of `HttpRequest` as the input and return an instance of `HttpResponse`. This is important in order to handle and represent generic HTTP operations. The only time that it won’t be returning `HttpResponse` would be when it’s raising exceptions (see Exception Handling).

The request downloader MUST resolve Location-based redirections when the HTTP method is not `HEAD`. In other words, for non-`HEAD` requests the returned `HttpResponse` must be the final response, after all redirects. For `HEAD` requests redirects MUST NOT be resolved.

Lastly, the request downloader function MUST support the `async/await` syntax.

## Exception Handling

Page Object developers could use the exception classes built inside web-poet to handle various ways additional requests MAY fail. In this section, we’ll see the rationale and ways the framework MUST be able to do that.

### Rationale

Frameworks that handle web-poet MUST be able to ensure that Page Objects having additional requests using `HttpClient` are able to work with any type of HTTP downloader implementation.

For example, in Python, the common HTTP libraries have different types of base exceptions when something has occurred:

> - aiohttp.ClientError

> - requests.RequestException

> - urllib.error.HTTPError

Imagine if Page Objects are expected to work in different backend implementations like the ones above, then it would cause the code to look like:

    import urllib

    import aiohttp
    import attrs
    import requests
    import web_poet
    from web_poet import validates_input

    @attrs.define
    class SomePage(web_poet.WebPage):
        http: web_poet.HttpClient

        @validates_input
        async def to_item(self):
            try:
                response = await self.http.get("...")
            except (aiohttp.ClientError, requests.RequestException, urllib.error.HTTPError):
                # handle the error here
                ...

Such code could turn messy in no time especially when the number of HTTP backends that Page Objects have to support are steadily increasing. Not to mention the plethora of exception types that HTTP libraries have. This means that Page Objects aren’t truly portable in different types of frameworks or environments. Rather, they’re only limited to work in the specific framework they’re supported.

In order for Page Objects to work in different Downloader Implementations, the framework that implements the HTTP Downloader backend MUST raise exceptions from the `web_poet.exceptions.http` module in lieu of the backend specific ones (e.g. aiohttp, requests, urllib, etc.).

This makes the code simpler:

    import attrs
    import web_poet
    from web_poet import validates_input

    @attrs.define
    class SomePage(web_poet.WebPage):
        http: web_poet.HttpClient

        @validates_input
        async def to_item(self):
            try:
                response = await self.http.get("...")
            except web_poet.exceptions.HttpError:
                # handle the error here
                ...

### Expected behavior for Exceptions

All exceptions that the HTTP Downloader Implementation (see Providing the Downloader doc section) explicitly raises when implementing it for web-poet MUST be `web_poet.exceptions.http.HttpError` (or a subclass from it).

For frameworks that implement and use web-poet, exceptions that occurred when handling the additional requests like connection errors, TLS errors, etc MUST be replaced by `web_poet.exceptions.http.HttpRequestError` by raising it explicitly.

For responses that are not really errors like in the `100-3xx` status code range, exception MUST NOT be raised at all. For responses with status codes in the `400-5xx` range, web-poet raises the `web_poet.exceptions.http.HttpResponseError` exception.

From this distinction, the framework MUST NOT raise `web_poet.exceptions.http.HttpResponseError` on its own at all, since the `HttpClient` already handles that.