What is the best web crawling software to use in a content discovery engine?

Victor Garcia April 5, 2013
Pinterest Stumbleupon Whatsapp

I’m creating an app similar to Stumbleupon with a content discovery engine. So what would be the most simple and efficient web crawling software to use to index mostly web pages, photos, and videos?

  1. Asya_K
    July 24, 2013 at 6:20 am

    google webmaster is best.. as far as i used...

  2. Chris Marcoe
    April 7, 2013 at 6:48 pm

    Hmmm...some good ideas here.

  3. salim benhouhou
    April 6, 2013 at 1:14 pm

    you can try teleport pro but it costs 49.95 $
    http://www.tenmax.com/teleport/pro/home.htm

    • Prasanna
      April 7, 2013 at 2:33 pm

      i agree with you mate, that one is good

  4. Alan Wade
    April 6, 2013 at 10:44 am

    Here is a list: http://web-crawler.software.informer.com/
    although it must be said that I have no knowledge at all of this type of software but worth a look, it may fill your needs.

  5. Varun Kuthe
    April 6, 2013 at 5:57 am

    Best website crawler is google webmaster. You can do everything in google webmaster. its not a software but more then a software.

Ads by Google