What are some essential PHP scripts for a webcrawler?

MafioSol November 9, 2011
Pinterest Stumbleupon Whatsapp

I’m creating a web crawler in PHP to produce urls from a web page. Building on this, what scripts do I need to have to:

Take a ‘root’ url embedded in the code,
Check to see if robot.txt exists at the root,
If robot.txt exists, read file and store information for future instructions.

 

 

Ads by Google