I have just read your PHP code in the article on how to build a basic web crawler. It is helpful because I’m doing my final project, which is basic web crawler in Java. But I’m still confused. Would you mind to help me? Really need your help.
If you're looking for a ready-made web crawler that's written in Java you might consider screen-scraper. I work for screen-scraper.com. Our software takes care of a ton of the heavy lifting that you would need to code yourself. It handles cookies, redirects, SSL, etc., etc.
Give it a try and let us know if you have any questions. screen-scraper.com
Make sure you read part 2 which may answer your question(s). I can help you with the PHP development, however I don't program in Java.
Sun (owner of Java) published an article titled "Writing a Web Crawler in the Java Programming Language" which may help you. There is also a number of open source public Java libraries which you can browse and get ideas from, such as Java Web Crawler, Niocchi and Crawler4j.
I was just about to post a link to that, thanks Jeffery.
Novi, I guess I should pipe up here as the author of that article, but it's been a long time since I touched Java at university - PHP defininitely more my thing too.
Read through those additional tuturials, then if you have a specific problem you can't solve, you could post your source in pastebin.com and attach the link to a new question, then maybe a fresh pair eye of programmers eyes could help you.
what exactly do you need help with?