Readers like you help support MUO. When you make a purchase using links on our site, we may earn an affiliate commission. Read More.
Can anyone explain with example? How can I correct my site? I am very confused. I do not understand what is this file, which Google does not find on my site.
2012-11-11 18:03:58
Josemon is right .. the google support itself having what is robots.txt file .. try googling it..
2012-11-06 01:49:24
robot.txt is a file which is scanned first by the webcrawlers, when they access a website ..in this file we define, which directories should be indexed and which are not ..it helps to reduce the bandwidth problems due to webcrawler access to our website...for more info http://www.webconfs.com/what-is-robots-txt-article-12.php visit this
2012-11-05 23:44:57
I assume you mean robot.txtIt's a file that tells search engine robots what information to gatherhttp://www.robotstxt.org/