Vivek Chidura asks:
What’s the best free website crawler software?
Comments for this Question are closed.
If you are looking for help, please ask a new question here.
We will be happy to help you!
4 Answers -
Pushpendra
May 1, 2012Best website crawler is google webmaster. You can do everything in google webmaster. its not a software but more then a software.
MetaDesign Solutions has vast experence in develop Adobe InDesign & InCopy Plug-ins for Mac and Windows. We develop performance boosting products & plug-ins for Adobe InDesign, Adobe InCopy, Quark & Apple’s products.
http://www.metadesignsolutions.com/products/products.php
Depends what you’re trying to do. So some examples:
Crawl to build a local search try http://www.wrensoft.com/ (free trial available)
To grab a complete copy (subject to some constraints) of a website http://www.httrack.com (free)
To build a sitemap.xml – http://www.xml-sitemaps.com/ (free version doesn’t go into subdirectories and max 500 pages)
Or do you mean a screen scraper which may be what Ray is looking for in his reply.
Or do you mean something else – like not a tool focused on a single website but more like Google’s web crawler that collects information on which to base search engine results – in that case best look on sourceforge
May 9, 2012Ray
May 1, 2012I was wondering if there are any good programming examples for this type of thing. I am writing a custom app to retrieve grocery prices for my Shopping App and having problems getting at the information…
Ray, I work for screen-scraper.com. With our software you can crawl and extract grocery prices from any number of Websites. You can also normalize the data and store it together in a single database.
Have a look over our features list and let us know if we can help.
http://screen-scraper.com/download/choose_version.php
June 19, 2012memo1288
May 9, 2012Well, I don’t know the best software, but at least I can give you a list:
http://www.snapfiles.com/freeware/downloader/fwoffline.html
It looks like the most popular program is WinHTTrack.
There are certainly better programs outside of the free category, and many of them offer a free trial. If you want to do a one-time job, they may be useful.
AIR
July 23, 2012Does anyone know of a good crawler or scraper that can extract email addresses from websites and put them in Excel?
Omg, a genius found a new way to harvest emails from websites ! way to go, spammer
October 17, 2012