I’m creating a web crawler in PHP to produce urls from a web page. Building on this, what scripts do I need to have to:
Take a ‘root’ url embedded in the code,
Check to see if robot.txt exists at the root,
If robot.txt exists, read file and store information for future instructions.