User-agent: * Disallow: /*notes/ Disallow: /dwsync.xml Disallow: /associates/ Disallow: /webcal/docs/ Disallow: /webcal/README_FILES/ Disallow: /webcal/tools/ Disallow: /webcal/ws/ Disallow: /zencart/admin/ Disallow: /zencart/bmz_cache/ Disallow: /zencart/download/ Disallow: /zencart/editors/ Disallow: /zencart/email/ Disallow: /zencart/extras/ Disallow: /zencart/includes/ Disallow: /zencart/temp/ Disallow: /zencart/fantversion.php Disallow: /zencart/error_log Disallow: /bugnuts.php Disallow: bugnuts.php Disallow: /.*error_log Disallow: /.*log Disallow: /sitemap_gen.py Sitemap: http://fabricland.com/sitemap.xml #Sitemap: http://fabricland.com/urllist.txt #The syntax of: #Disallow: /directory/ #Is 100% the appropriate way of disallowing a directory, the alternate method of. # #Disallow: /directory #Will disallow the directory as well as any pages of the same name. # #A request for example.com/directory will be forwarded to example.com/directory/index.ext which is the true url. Regardless of whether the request was made to /directory if we have disallowed /directory/ then the robot should not index this link based on the standard if that bot follows robots.txt. #I understand that robots still do but that is the fault of the bot in question. The trailing slash is the appropriate way to disallow a directory in its entirety and only that directory.