Readers like you help support MUO. When you make a purchase using links on our site, we may earn an affiliate commission. Read More.
I'm creating a web crawler in PHP to produce urls from a web page. Building on this, what scripts do I need to have to:
Take a 'root' url embedded in the code,
Check to see if robot.txt exists at the root,
If robot.txt exists, read file and store information for future instructions.
2011-11-10 10:24:00
Hi Mafio - I'm not really sure what you're asking for. The pseudo code you have there is a good start certainly, now you just need to code that. Read those links Jay posted above then ask again with something more specific.
2011-11-10 05:43:00
may be, articles of James Bruce and James Bruce himself can help you !Read his articles:https://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-websitehttps://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-website-2/