What’s the best free website crawler software?
Question by Vivek Chidura /
Ads by Google

What’s the best free website crawler software?

Browse other questions & answers in the category ; tagged , , .
Join live MakeUseOf Groups on Grouvi App Join live Groups on Grouvi
Awesome Internet (and Tips)
Awesome Internet (and Tips)
1420 Members
Stay Invisible Online
Stay Invisible Online
1036 Members
Deep Web Links & Tips
Deep Web Links & Tips
700 Members
Web Security & Privacy
Web Security & Privacy
510 Members
Music Apps & Sites
Music Apps & Sites
393 Members

Comments for this Question are closed.

Answers (7)
  • AIR

    Does anyone know of a good crawler or scraper that can extract email addresses from websites and put them in Excel?

    • air_spammer

      Omg, a genius found a new way to harvest emails from websites ! way to go, spammer

  • memo1288

    Well, I don't know the best software, but at least I can give you a list:
    It looks like the most popular program is WinHTTrack.
    There are certainly better programs outside of the free category, and many of them offer a free trial. If you want to do a one-time job, they may be useful.

  • Ray

    I was wondering if there are any good programming examples for this type of thing. I am writing a custom app to retrieve grocery prices for my Shopping App and having problems getting at the information...

    • Scott Wilson

      Ray, I work for screen-scraper.com. With our software you can crawl and extract grocery prices from any number of Websites. You can also normalize the data and store it together in a single database.

      Have a look over our features list and let us know if we can help.


  • Pushpendra

    Best website crawler is google webmaster. You can do everything in google webmaster. its not a software but more then a software.

    MetaDesign Solutions has vast experence in develop Adobe InDesign & InCopy Plug-ins for Mac and Windows. We develop performance boosting products & plug-ins for Adobe InDesign, Adobe InCopy, Quark & Apple's products.


    • rob

      Depends what you're trying to do. So some examples:

      Crawl to build a local search try http://www.wrensoft.com/ (free trial available)

      To grab a complete copy (subject to some constraints) of a website http://www.httrack.com (free)

      To build a sitemap.xml - http://www.xml-sitemaps.com/ (free version doesn't go into subdirectories and max 500 pages)

      Or do you mean a screen scraper which may be what Ray is looking for in his reply.

      Or do you mean something else - like not a tool focused on a single website but more like Google's web crawler that collects information on which to base search engine results - in that case best look on sourceforge

Ads by Google
New comment

Please login to avoid entering captcha

Log In