| Extracted from Pike v7.8 release 866 at 2018-07-18.

[Top]
Web
Web.Crawler
Web.Crawler.Policy

# Class Web.Crawler.Policy

Description

  The crawler policy object.

Variable max_concurrent_fetchers

  `int Web.Crawler.Policy.max_concurrent_fetchers`

Description

  Maximum number of fetchers. Defaults to 100.

Variable max_bits_per_second_total

  `int Web.Crawler.Policy.max_bits_per_second_total`

Description

  Maximum number of bits per second. Defaults to off (0).

Variable max_bits_per_second_per_host

  `int Web.Crawler.Policy.max_bits_per_second_per_host`

Description

  Maximum number of bits per second, per host. Defaults to off (0).

Variable bandwidth_throttling_floating_window_width

  `int Web.Crawler.Policy.bandwidth_throttling_floating_window_width`

Description

  Bandwidth throttling floating window width. Defaults to 30.

Variable max_concurrent_fetchers_per_host

  `int Web.Crawler.Policy.max_concurrent_fetchers_per_host`

Description

  Maximum concurrent fetchers per host. Defaults to 1.

Variable min_delay_per_host

  `int Web.Crawler.Policy.min_delay_per_host`

Description

  Minimum delay per host. Defaults to 0.