# 7/8/14 - Nate #google webmaster tools said googlebot was blocked from brewersassociation.org. Nate commented out these custom rules to try and fix. # This file is read and obeyed by polite search engines # Please refer to the W3C notes: # http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 # or http://info.webcrawler.com/mak/projects/robots/exclusion.html *User-agent: * # Added by Jim 2007-04-16 to (hopefully) decrease server loads in the morning # time is in GMT, works out noon-4 AM MST # This is from a proposed Extended Standard for Robot Exclusion, not widely adopted yet. # Who knows though, maybe some crawlers will honor it. *Visit-time: 1800-1000 # # Added by support@evolutionhosting.com to slow down GoogleBot *Crawl-delay: 10 *Request-rate: 1/6 # Turn away unwanted search engines # Rover is a bad dog User-agent: Webdup/0.9 Disallow: /