What’s the best free website crawler software?
Question by Vivek Chidura /

What’s the best free website crawler software?

Browse other questions & answers in the category ; tagged , , ; or ask your own question.

Comments for this Question are closed.

If you are looking for help, please ask a new question.

We will be happy to help you!

Answers (7)
  • AIR

    Does anyone know of a good crawler or scraper that can extract email addresses from websites and put them in Excel?

    • air_spammer

      Omg, a genius found a new way to harvest emails from websites ! way to go, spammer

  • memo1288

    Well, I don’t know the best software, but at least I can give you a list:
    http://www.snapfiles.com/freeware/downloader/fwoffline.html
    It looks like the most popular program is WinHTTrack.
    There are certainly better programs outside of the free category, and many of them offer a free trial. If you want to do a one-time job, they may be useful.

  • Ray

    I was wondering if there are any good programming examples for this type of thing. I am writing a custom app to retrieve grocery prices for my Shopping App and having problems getting at the information…

    • Scott Wilson

      Ray, I work for screen-scraper.com. With our software you can crawl and extract grocery prices from any number of Websites. You can also normalize the data and store it together in a single database.

      Have a look over our features list and let us know if we can help.

      http://screen-scraper.com/download/choose_version.php

  • Pushpendra

    Best website crawler is google webmaster. You can do everything in google webmaster. its not a software but more then a software.

    MetaDesign Solutions has vast experence in develop Adobe InDesign & InCopy Plug-ins for Mac and Windows. We develop performance boosting products & plug-ins for Adobe InDesign, Adobe InCopy, Quark & Apple’s products.

    http://www.metadesignsolutions.com/products/products.php

    • rob

      Depends what you’re trying to do. So some examples:

      Crawl to build a local search try http://www.wrensoft.com/ (free trial available)

      To grab a complete copy (subject to some constraints) of a website http://www.httrack.com (free)

      To build a sitemap.xml – http://www.xml-sitemaps.com/ (free version doesn’t go into subdirectories and max 500 pages)

      Or do you mean a screen scraper which may be what Ray is looking for in his reply.

      Or do you mean something else – like not a tool focused on a single website but more like Google’s web crawler that collects information on which to base search engine results – in that case best look on sourceforge