How can I make my PHP web crawler automatically skill broken links?

Jerome Yurow December 17, 2011
Pinterest Stumbleupon Whatsapp

I have a list of links on a website that I wish to check with a PHP web crawler program. Some of these links may be broken. Rather than attempting to follow a broken link and getting, say, a 404 error message, I would like to skip over the link before trying to load it.

I am using the features of simple_html_dom.php in my web crawler. So I would like to detect a broken link before I perform $html->load_file($link);

How can I do this?

Ads by Google