A robots.txt file is a short text file that resides in your home directory. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see.
This tool will make your robots.txt file. As a little 'extra' it also includes a block to many unwanted spiders that only crawl your site in order to collect the email addresses stored on your pages.(SPAM-bots).
[ Robots.txt Generator ]
Assinar:
Postar comentários (Atom)
Nenhum comentário:
Postar um comentário