# Signals

Scrapy uses signals extensively to notify when certain events occur. You can catch some of those signals in your Scrapy project (using an extension, for example) to perform additional tasks or extend Scrapy to add functionality not provided out of the box.

Even though signals provide several arguments, the handlers that catch them don’t need to accept all of them - the signal dispatching mechanism will only deliver the arguments that the handler receives.

You can connect to signals (or send your own) through the Signals API.

Here is a simple example showing how you can catch signals and perform some action:

    from scrapy import signals
    from scrapy import Spider

    class DmozSpider(Spider):
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
        ]

        @classmethod
        def from_crawler(cls, crawler, *args, **kwargs):
            spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)
            crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
            return spider

        def spider_closed(self, spider):
            spider.logger.info("Spider closed: %s", spider.name)

        def parse(self, response):
            pass

## Deferred signal handlers

Some signals support returning `Deferred` or awaitable objects from their handlers, allowing you to run asynchronous code that does not block Scrapy. If a signal handler returns one of these objects, Scrapy waits for that asynchronous operation to finish.

Let’s take an example using coroutines:

    import scrapy

    class SignalSpider(scrapy.Spider):
        name = "signals"
        start_urls = ["https://quotes.toscrape.com/page/1/"]

        @classmethod
        def from_crawler(cls, crawler, *args, **kwargs):
            spider = super(SignalSpider, cls).from_crawler(crawler, *args, **kwargs)
            crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)
            return spider

        async def item_scraped(self, item):
            # Send the scraped item to the server
            response = await treq.post(
                "http://example.com/post",
                json.dumps(item).encode("ascii"),
                headers={b"Content-Type": [b"application/json"]},
            )

            return response

        def parse(self, response):
            for quote in response.css("div.quote"):
                yield {
                    "text": quote.css("span.text::text").get(),
                    "author": quote.css("small.author::text").get(),
                    "tags": quote.css("div.tags a.tag::text").getall(),
                }

See the Built-in signals reference below to know which signals support `Deferred` and awaitable objects.

## Built-in signals reference

Here’s the list of Scrapy built-in signals and their meaning.

### Engine signals

#### engine_started

scrapy.signals.engine_started()

  Sent when the Scrapy engine has started crawling.

  This signal supports returning deferreds from its handlers.

Note

This signal may be fired after the `spider_opened` signal, depending on how the spider was started. So don’t rely on this signal getting fired before `spider_opened`.

#### engine_stopped

scrapy.signals.engine_stopped()

  Sent when the Scrapy engine is stopped (for example, when a crawling process has finished).

  This signal supports returning deferreds from its handlers.

### Item signals

Note

As at max `CONCURRENT_ITEMS` items are processed in parallel, many deferreds are fired together using `DeferredList`. Hence the next batch waits for the `DeferredList` to fire and then runs the respective item signal handler for the next batch of scraped items.

#### item_scraped

scrapy.signals.item_scraped(item, response, spider)

  Sent when an item has been scraped, after it has passed all the Item Pipeline stages (without being dropped).

  This signal supports returning deferreds from its handlers.

  Parameters:

    - item (item object) – the scraped item

    - spider (`Spider` object) – the spider which scraped the item

    - response (`Response` object) – the response from where the item was scraped

#### item_dropped

scrapy.signals.item_dropped(item, response, exception, spider)

  Sent after an item has been dropped from the Item Pipeline when some stage raised a `DropItem` exception.

  This signal supports returning deferreds from its handlers.

  Parameters:

    - item (item object) – the item dropped from the Item Pipeline

    - spider (`Spider` object) – the spider which scraped the item

    - response (`Response` object) – the response from where the item was dropped

    - exception (`DropItem` exception) – the exception (which must be a `DropItem` subclass) which caused the item to be dropped

#### item_error

scrapy.signals.item_error(item, response, spider, failure)

  Sent when a Item Pipeline generates an error (i.e. raises an exception), except `DropItem` exception.

  This signal supports returning deferreds from its handlers.

  Parameters:

    - item (item object) – the item that caused the error in the Item Pipeline

    - response (`Response` object) – the response being processed when the exception was raised

    - spider (`Spider` object) – the spider which raised the exception

    - failure (twisted.python.failure.Failure) – the exception raised

### Spider signals

#### spider_closed

scrapy.signals.spider_closed(spider, reason)

  Sent after a spider has been closed. This can be used to release per-spider resources reserved on `spider_opened`.

  This signal supports returning deferreds from its handlers.

  Parameters:

    - spider (`Spider` object) – the spider which has been closed

    - reason (str) – a string which describes the reason why the spider was closed. If it was closed because the spider has completed scraping, the reason is `'finished'`. Otherwise, if the spider was manually closed by calling the `close_spider` engine method, then the reason is the one passed in the `reason` argument of that method (which defaults to `'cancelled'`). If the engine was shutdown (for example, by hitting Ctrl-C to stop it) the reason will be `'shutdown'`.

#### spider_opened

scrapy.signals.spider_opened(spider)

  Sent after a spider has been opened for crawling. This is typically used to reserve per-spider resources, but can be used for any task that needs to be performed when a spider is opened.

  This signal supports returning deferreds from its handlers.

  Parameters:

    spider (`Spider` object) – the spider which has been opened

#### spider_idle

scrapy.signals.spider_idle(spider)

  Sent when a spider has gone idle, which means the spider has no further:

  > - requests waiting to be downloaded

  > - requests scheduled

  > - items being processed in the item pipeline

  If the idle state persists after all handlers of this signal have finished, the engine starts closing the spider. After the spider has finished closing, the `spider_closed` signal is sent.

  You may raise a `DontCloseSpider` exception to prevent the spider from being closed.

  Alternatively, you may raise a `CloseSpider` exception to provide a custom spider closing reason. An idle handler is the perfect place to put some code that assesses the final spider results and update the final closing reason accordingly (e.g. setting it to ‘too_few_results’ instead of ‘finished’).

  This signal does not support returning deferreds from its handlers.

  Parameters:

    spider (`Spider` object) – the spider which has gone idle

Note

Scheduling some requests in your `spider_idle` handler does not guarantee that it can prevent the spider from being closed, although it sometimes can. That’s because the spider may still remain idle if all the scheduled requests are rejected by the scheduler (e.g. filtered due to duplication).

#### spider_error

scrapy.signals.spider_error(failure, response, spider)

  Sent when a spider callback generates an error (i.e. raises an exception).

  This signal does not support returning deferreds from its handlers.

  Parameters:

    - failure (twisted.python.failure.Failure) – the exception raised

    - response (`Response` object) – the response being processed when the exception was raised

    - spider (`Spider` object) – the spider which raised the exception

#### feed_slot_closed

scrapy.signals.feed_slot_closed(slot)

  Sent when a feed exports slot is closed.

  This signal supports returning deferreds from its handlers.

  Parameters:

    slot (scrapy.extensions.feedexport.FeedSlot) – the slot closed

#### feed_exporter_closed

scrapy.signals.feed_exporter_closed()

  Sent when the feed exports extension is closed, during the handling of the `spider_closed` signal by the extension, after all feed exporting has been handled.

  This signal supports returning deferreds from its handlers.

### Request signals

#### request_scheduled

scrapy.signals.request_scheduled(request, spider)

  Sent when the engine schedules a `Request`, to be downloaded later.

  This signal does not support returning deferreds from its handlers.

  Parameters:

    - request (`Request` object) – the request that reached the scheduler

    - spider (`Spider` object) – the spider that yielded the request

#### request_dropped

scrapy.signals.request_dropped(request, spider)

  Sent when a `Request`, scheduled by the engine to be downloaded later, is rejected by the scheduler.

  This signal does not support returning deferreds from its handlers.

  Parameters:

    - request (`Request` object) – the request that reached the scheduler

    - spider (`Spider` object) – the spider that yielded the request

#### request_reached_downloader

scrapy.signals.request_reached_downloader(request, spider)

  Sent when a `Request` reached downloader.

  This signal does not support returning deferreds from its handlers.

  Parameters:

    - request (`Request` object) – the request that reached downloader

    - spider (`Spider` object) – the spider that yielded the request

#### request_left_downloader

scrapy.signals.request_left_downloader(request, spider)

  New in version 2.0.

  Sent when a `Request` leaves the downloader, even in case of failure.

  This signal does not support returning deferreds from its handlers.

  Parameters:

    - request (`Request` object) – the request that reached the downloader

    - spider (`Spider` object) – the spider that yielded the request

#### bytes_received

New in version 2.2.

scrapy.signals.bytes_received(data, request, spider)

  Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is received for a specific request. This signal might be fired multiple times for the same request, with partial data each time. For instance, a possible scenario for a 25 kb response would be two signals fired with 10 kb of data, and a final one with 5 kb of data.

  Handlers for this signal can stop the download of a response while it is in progress by raising the `StopDownload` exception. Please refer to the Stopping the download of a Response topic for additional information and examples.

  This signal does not support returning deferreds from its handlers.

  Parameters:

    - data (`bytes` object) – the data received by the download handler

    - request (`Request` object) – the request that generated the download

    - spider (`Spider` object) – the spider associated with the response

#### headers_received

New in version 2.5.

scrapy.signals.headers_received(headers, body_length, request, spider)

  Sent by the HTTP 1.1 and S3 download handlers when the response headers are available for a given request, before downloading any additional content.

  Handlers for this signal can stop the download of a response while it is in progress by raising the `StopDownload` exception. Please refer to the Stopping the download of a Response topic for additional information and examples.

  This signal does not support returning deferreds from its handlers.

  Parameters:

    - headers (`scrapy.http.headers.Headers` object) – the headers received by the download handler

    - body_length (int) – expected size of the response body, in bytes

    - request (`Request` object) – the request that generated the download

    - spider (`Spider` object) – the spider associated with the response

### Response signals

#### response_received

scrapy.signals.response_received(response, request, spider)

  Sent when the engine receives a new `Response` from the downloader.

  This signal does not support returning deferreds from its handlers.

  Parameters:

    - response (`Response` object) – the response received

    - request (`Request` object) – the request that generated the response

    - spider (`Spider` object) – the spider for which the response is intended

Note

The `request` argument might not contain the original request that reached the downloader, if a Downloader Middleware modifies the `Response` object and sets a specific `request` attribute.

#### response_downloaded

scrapy.signals.response_downloaded(response, request, spider)

  Sent by the downloader right after a `HTTPResponse` is downloaded.

  This signal does not support returning deferreds from its handlers.

  Parameters:

    - response (`Response` object) – the response downloaded

    - request (`Request` object) – the request that generated the response

    - spider (`Spider` object) – the spider for which the response is intended