Readers like you help support MUO. When you make a purchase using links on our site, we may earn an affiliate commission. Read More.

I'm creating a web crawler in PHP to produce urls from a web page. Building on this, what scripts do I need to have to:

Take a 'root' url embedded in the code,

Check to see if robot.txt exists at the root,

If robot.txt exists, read file and store information for future instructions.

 

 

James Bruce
2011-11-10 10:24:00
Hi Mafio - I'm not really sure what you're asking for. The pseudo code you have there is a good start certainly, now you just need to code that. Read those links Jay posted above then ask again with something more specific.
MAKEUSEOF VIDEO OF THE DAY
SCROLL TO CONTINUE WITH CONTENT
Jay
2011-11-10 05:43:00
may be, articles of James Bruce and James Bruce himself can help you !Read his articles:https://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-websitehttps://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-website-2/