Package logsparser :: Package extras :: Module robots

[frames] | no frames]

# Module robots

source code

In this module we define a regular expression used to fetch the most common robots.

Variables
| robots = ``[``'``antibot``'``,` `'``appie``'``,` `'``architext``'``,` `'``bingbot``'``,` `'``bjaalan``...``
| robot_regex = `re.compile(r'`(?i)`antibot`|`appie`|`architext`|`bingbot`...``
| __package__ = ``'``logsparser.extras``'``

Variables Details

### robots

Value:

      ['antibot',
       'appie',
       'architext',
       'bingbot',
       'bjaaland',
       'digout4u',
       'echo',
       'fast-webcrawler',
      ...

### robot_regex

Value:

      re.compile(r'(?i)antibot|appie|architext|bingbot|bjaaland|digout4u|ech
      o|fast-webcrawler|ferret|googlebot|gulliver|harvest|htdig|ia_archiver|
      askjeeves|jennybot|linkwalker|lycos|mercator|moget|muscatferret|myweb|
      netcraft|nomad|petersnews|scooter|slurp|unlost_web_crawler|voila|voyag
      er|webbase|weblayers|wisenutbot|aport|awbot|baiduspider|bobby|boris|bu
      mblebee|cscrawler|daviesbot|exactseek|ezresult|gigabot|gnodspider|grub
      |henrythemiragorobot|holmes|internetseer|justview|linkbot|metager-link
      checker|linkchecker|microsoft_url_control|msiecrawler|nagios|perman|po
      ...