Readers like you help support MUO. When you make a purchase using links on our site, we may earn an affiliate commission. Read More.

I have just read your PHP code in the article on how to build a basic web crawler. It is helpful because I'm doing my final project, which is basic web crawler in Java. But I'm still confused. Would you mind to help me? Really need your help.

Scott Wilson
2012-07-12 21:40:00
novi,If you're looking for a ready-made web crawler that's written in Java you might consider screen-scraper. I work for screen-scraper.com. Our software takes care of a ton of the heavy lifting that you would need to code yourself. It handles cookies, redirects, SSL, etc., etc. Give it a try and let us know if you have any questions. screen-scraper.com-Scott
MAKEUSEOF VIDEO OF THE DAY
SCROLL TO CONTINUE WITH CONTENT
Jeffery Fabish
2011-09-22 19:40:00
Novi,Make sure you read part 2 which may answer your question(s). I can help you with the PHP development, however I don't program in Java. Sun (owner of Java) published an article titled " Writing a Web Crawler in the Java Programming Language " which may help you. There is also a number of open source public Java libraries which you can browse and get ideas from, such as Java Web Crawler , Niocchi and Crawler4j .
James Bruce
2011-09-23 07:17:00
I was just about to post a link to that, thanks Jeffery. Novi, I guess I should pipe up here as the author of that article, but it's been a long time since I touched Java at university - PHP defininitely more my thing too. Read through those additional tuturials, then if you have a specific problem you can't solve, you could post your source in pastebin.com and attach the link to a new question, then maybe a fresh pair eye of programmers eyes could help you. 
Tina
2011-09-22 19:02:00
Novi,what exactly do you need help with?