# /robots.txt file for http://webcrawler.com/




# mail webmaster@webcrawler.com for constructive criticism









User-agent: *




Disallow: /tmp




Disallow: /logs




Disallow: /download




Disalow: /_private




Disalow: /images




Disalow: /PDFs




Disalow: / _THEMES




Disalow: / _vti_log




Disalow: / _BORDERS




Disalow: /BARS




Disalow: /_aspnet_client




Disalow: /_OVERLAY




Disalow: /_derived




Disalow: /_FPCLASS









# The first two lines, starting with '#', specify a comment 




# The first paragraph specifies that the robot called 'webcrawler' has nothing disallowed: it may go anywhere. 




# The second paragraph indicates that the robot called 'lycra' has all relative URLs starting with '/' disallowed. Because all  relative URL's on a server start with '/', this means the entire site is closed off. 




# The third paragraph indicates that all other robots should not visit URLs starting with /tmp or /log. Note the '*' is a special token, meaning "any other User-agent"; you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines.