Readers like you help support MUO. When you make a purchase using links on our site, we may earn an affiliate commission. Read More.
I have just read your PHP code in the article on how to build a basic web crawler. It is helpful because I'm doing my final project, which is basic web crawler in Java. But I'm still confused. Would you mind to help me? Really need your help.
2012-07-12 21:40:00
novi,If you're looking for a ready-made web crawler that's written in Java you might consider screen-scraper. I work for screen-scraper.com. Our software takes care of a ton of the heavy lifting that you would need to code yourself. It handles cookies, redirects, SSL, etc., etc. Give it a try and let us know if you have any questions. screen-scraper.com-Scott
2011-09-22 19:40:00
Novi,Make sure you read
part 2
which may answer your question(s). I can help you with the PHP development, however I don't program in Java. Sun (owner of Java) published an article titled "
Writing a Web Crawler in the Java Programming Language
" which may help you. There is also a number of open source public Java libraries which you can browse and get ideas from, such as
Java Web Crawler
,
Niocchi
and
Crawler4j
.
2011-09-23 07:17:00
I was just about to post a link to that, thanks Jeffery. Novi, I guess I should pipe up here as the author of that article, but it's been a long time since I touched Java at university - PHP defininitely more my thing too. Read through those additional tuturials, then if you have a specific problem you can't solve, you could post your source in pastebin.com and attach the link to a new question, then maybe a fresh pair eye of programmers eyes could help you.
2011-09-22 19:02:00
Novi,what exactly do you need help with?