What’s the best free website crawler software?
Does anyone know of a good crawler or scraper that can extract email addresses from websites and put them in Excel?
Omg, a genius found a new way to harvest emails from websites ! way to go, spammer
Well, I don't know the best software, but at least I can give you a list:
It looks like the most popular program is WinHTTrack.
There are certainly better programs outside of the free category, and many of them offer a free trial. If you want to do a one-time job, they may be useful.
I was wondering if there are any good programming examples for this type of thing. I am writing a custom app to retrieve grocery prices for my Shopping App and having problems getting at the information...
Ray, I work for screen-scraper.com. With our software you can crawl and extract grocery prices from any number of Websites. You can also normalize the data and store it together in a single database.
Have a look over our features list and let us know if we can help.
Best website crawler is google webmaster. You can do everything in google webmaster. its not a software but more then a software.
MetaDesign Solutions has vast experence in develop Adobe InDesign & InCopy Plug-ins for Mac and Windows. We develop performance boosting products & plug-ins for Adobe InDesign, Adobe InCopy, Quark & Apple's products.
Depends what you're trying to do. So some examples:
Crawl to build a local search try http://www.wrensoft.com/ (free trial available)
To grab a complete copy (subject to some constraints) of a website http://www.httrack.com (free)
To build a sitemap.xml - http://www.xml-sitemaps.com/ (free version doesn't go into subdirectories and max 500 pages)
Or do you mean a screen scraper which may be what Ray is looking for in his reply.
Or do you mean something else - like not a tool focused on a single website but more like Google's web crawler that collects information on which to base search engine results - in that case best look on sourceforge