I’m creating an app similar to Stumbleupon with a content discovery engine. So what would be the most simple and efficient web crawling software to use to index mostly web pages, photos, and videos?
google webmaster is best.. as far as i used...
Hmmm...some good ideas here.
you can try teleport pro but it costs 49.95 $
i agree with you mate, that one is good
Here is a list: http://web-crawler.software.informer.com/
although it must be said that I have no knowledge at all of this type of software but worth a look, it may fill your needs.
Best website crawler is google webmaster. You can do everything in google webmaster. its not a software but more then a software.