What’s the best free website crawler software?

Vivek Chidura April 30, 2012

What’s the best free website crawler software?

  1. AIR
    July 23, 2012 at 10:15 pm

    Does anyone know of a good crawler or scraper that can extract email addresses from websites and put them in Excel?

    • air_spammer
      October 17, 2012 at 2:41 pm

      Omg, a genius found a new way to harvest emails from websites ! way to go, spammer

  2. memo1288
    May 9, 2012 at 5:15 pm

    Well, I don't know the best software, but at least I can give you a list:
    It looks like the most popular program is WinHTTrack.
    There are certainly better programs outside of the free category, and many of them offer a free trial. If you want to do a one-time job, they may be useful.

  3. Ray
    May 1, 2012 at 7:31 pm

    I was wondering if there are any good programming examples for this type of thing. I am writing a custom app to retrieve grocery prices for my Shopping App and having problems getting at the information...

    • Scott Wilson
      June 19, 2012 at 9:33 pm

      Ray, I work for screen-scraper.com. With our software you can crawl and extract grocery prices from any number of Websites. You can also normalize the data and store it together in a single database.

      Have a look over our features list and let us know if we can help.


  4. Pushpendra
    May 1, 2012 at 9:24 am

    Best website crawler is google webmaster. You can do everything in google webmaster. its not a software but more then a software.

    MetaDesign Solutions has vast experence in develop Adobe InDesign & InCopy Plug-ins for Mac and Windows. We develop performance boosting products & plug-ins for Adobe InDesign, Adobe InCopy, Quark & Apple's products.


    • rob
      May 9, 2012 at 12:49 pm

      Depends what you're trying to do. So some examples:

      Crawl to build a local search try http://www.wrensoft.com/ (free trial available)

      To grab a complete copy (subject to some constraints) of a website http://www.httrack.com (free)

      To build a sitemap.xml - http://www.xml-sitemaps.com/ (free version doesn't go into subdirectories and max 500 pages)

      Or do you mean a screen scraper which may be what Ray is looking for in his reply.

      Or do you mean something else - like not a tool focused on a single website but more like Google's web crawler that collects information on which to base search engine results - in that case best look on sourceforge

Ads by Google