Can you help me with a Basic Web Crawler project coded in Java?

novi September 22, 2011

I have just read your PHP code in the article on how to build a basic web crawler. It is helpful because I’m doing my final project, which is basic web crawler in Java. But I’m still confused. Would you mind to help me? Really need your help.

  1. Scott Wilson
    July 12, 2012 at 9:40 pm


    If you're looking for a ready-made web crawler that's written in Java you might consider screen-scraper. I work for Our software takes care of a ton of the heavy lifting that you would need to code yourself. It handles cookies, redirects, SSL, etc., etc.

    Give it a try and let us know if you have any questions.


  2. Jeffery Fabish
    September 22, 2011 at 7:40 pm


    Make sure you read part 2 which may answer your question(s). I can help you with the PHP development, however I don't program in Java.

    Sun (owner of Java) published an article titled "Writing a Web Crawler in the Java Programming Language" which may help you. There is also a number of open source public Java libraries which you can browse and get ideas from, such as Java Web Crawler, Niocchi and Crawler4j.

    • James Bruce
      September 23, 2011 at 7:17 am

      I was just about to post a link to that, thanks Jeffery. 

      Novi, I guess I should pipe up here as the author of that article, but it's been a long time since I touched Java at university - PHP defininitely more my thing too. 

      Read through those additional tuturials, then if you have a specific problem you can't solve, you could post your source in and attach the link to a new question, then maybe a fresh pair eye of programmers eyes could help you. 

  3. Tina
    September 22, 2011 at 7:02 pm


    what exactly do you need help with?