How can I build a basic web crawler?

jayachandran November 12, 2011

This question is specially for James Bruce who made me think that designing crawler is easy. Now please help me with the crawler code.

I tried to implement the code in my Blogger (?), but it didn’t work. Can anyone please help me with this? The code is here: How To Build A Basic Web Crawler To Pull Information From A Website (Part 2) How To Build A Basic Web Crawler To Pull Information From A Website (Part 2) Read More

  1. Anonymous
    December 7, 2011 at 4:19 am

    can i create search engine of my local region,with that open source code ,Please guide me in this.. 

  2. James Bruce
    November 13, 2011 at 9:52 am

    Hi jayachandran. The article in question refers quite clearly to part 1 also, in which the requirements clearly state

    "you will need a server to run PHP. You have a number of options here:
    If you host your own blog using WordPress, you already have one, so upload the files you write via FTP and run them from there. Matt showed us some free FTP clients for Windows you could use.

    If you don’t have a web server but do have an old PC sitting around, then you could follow Dave’s tutorial here to turn an old PC into a web server.

    Just one computer? Don’t worry – Jeffry showed us how we can run a local server inside of Windows or Mac."


    Unfortunately, Blogger doesn't let you use PHP, hence why it wasn't listed as a supported method (and neither is a free blog). You need something which is self-hosted on your own purchased webspace - no free blog solution is going to allow to run scripting code in your pages I'm afraid.

  3. Jeff Fabish
    November 12, 2011 at 6:22 pm

    Hi Jayachandran,

    Firstly, does Blogger even allow you to embed PHP in your template? 

    Make sure that you read part one. This article isn't geared towards an absolute beginner, you should probably get a fundamental understanding of the language, before you go out and try to be a big hit. If you're just looking for a web crawler, there are plenty of open source or otherwise free web crawlers you can implement without having to delve into code. 

    Learning PHP
    PHP.Net - PHP Tutorial
    Tizag - PHP Tutorial
    W3 Schools - PHP Tutorial
    Zend - PHP Tutorial