Pinterest Stumbleupon Whatsapp
Ads by Google

No, it’s not Spiderman’s latest web slinging tool but something that’s more real world. Like the World Wide Web.

The Invisible Web refers to the part of the WWW that’s not indexed by the search engines. Most of us think that that search powerhouses like Google and Bing are like the Great Oracle”¦they see everything. Unfortunately, they can’t because they aren’t divine at all; they are just web spiders who index pages by following one hyperlink after the other.

But there are some places where a spider cannot enter. Take library databases which need a password for access. Or even pages that belong to private networks of organizations. Dynamically generated web pages in response to a query are often left un-indexed by search engine spiders.

Search engine technology has progressed by leaps and bounds. Today, we have real time search and the capability to index Flash based and PDF content. Even then, there remain large swathes of the web which a general search engine cannot penetrate. The term, Deep Net, Deep Web or Invisible Web lingers on.

To get a more precise idea of the nature of this ‘Dark Continent’ involving the invisible and web search engines, read what Wikipedia has to say about the Deep Web. The figures are attention grabbers – the size of the open web is 167 terabytes. The Invisible Web is estimated at 91,000 terabytes. Check this out – the Library of Congress, in 1997, was figured to have close to 3,000 terabytes!

How do we get to this mother load of information?

Ads by Google

That’s what this post is all about. Let’s get to know a few resources which will be our deep diving vessel for the Invisible Web. Some of these are invisible web search engines with specifically indexed information.

Infomine

invisible web search engines

Infomine has been built by a pool of libraries in the United States. Some of them are University of California, Wake Forest University, California State University, and the University of Detroit. Infomine ‘mines’ information from databases, electronic journals, electronic books, bulletin boards, mailing lists, online library card catalogs, articles, directories of researchers, and many other resources.

You can search by subject category and further tweak your search using the search options. Infomine is not only a standalone search engine for the Deep Web but also a staging point for a lot of other reference information. Check out its Other Search Tools and General Reference links at the bottom.

The WWW Virtual Library

invisible web search engines

This is considered to be the oldest catalog on the web and was started by started by Tim Berners-Lee, the creator of the web. So, isn’t it strange that it finds a place in the list of Invisible Web resources? Maybe, but the WWW Virtual Library lists quite a lot of relevant resources on quite a lot of subjects. You can go vertically into the categories or use the search bar. The screenshot shows the alphabetical arrangement of subjects covered at the site.

Intute

invisible web search engines

Intute is UK centric, but it has some of the most esteemed universities of the region providing the resources for study and research. You can browse by subject or do a keyword search for academic topics like agriculture to veterinary medicine. The online service has subject specialists who review and index other websites that cater to the topics for study and research.

Intute also provides free of cost over 60 free online tutorials to learn effective internet research skills. Tutorials are step by step guides and are arranged around specific subjects.

Complete Planet

search invisible web

Complete Planet calls itself the ‘front door to the Deep Web’. This free and well designed directory resource makes it easy to access the mass of dynamic databases that are cloaked from a general purpose search. The databases indexed by Complete Planet number around 70,000 and range from Agriculture to Weather. Also thrown in are databases like Food & Drink and Military.

For a really effective Deep Web search, try out the Advanced Search options where among other things, you can set a date range.

Infoplease

search invisible web

Infoplease is an information portal with a host of features. Using the site, you can tap into a good number of encyclopedias, almanacs, an atlas, and biographies. Infoplease also has a few nice offshoots like Factmonster.com for kids and Biosearch, a search engine just for biographies.

DeepPeep

search invisible web

DeepPeep aims to enter the Invisible Web through forms that query databases and web services for information. Typed queries open up dynamic but short lived results which cannot be indexed by normal search engines. By indexing databases, DeepPeep hopes to track 45,000 forms across 7 domains.

The domains covered by DeepPeep (Beta) are Auto, Airfare, Biology, Book, Hotel, Job, and Rental. Being a beta service, there are occasional glitches as some results don’t load in the browser.

IncyWincy

how to use the invisible web

IncyWincy is an Invisible Web search engine and it behaves as a meta-search engine by tapping into other search engines and filtering the results. It searches the web, directory, forms, and images. With a free registration, you can track search results with alerts.

DeepWebTech

how to use the invisible web

DeepWebTech gives you five search engines (and browser plugins) for specific topics. The search engines cover science, medicine, and business. Using these topic specific search engines, you can query the underlying databases in the Deep Web.

Scirus

how to use the invisible web

Scirus has a pure scientific focus. It is a far reaching research engine that can scour journals, scientists’ homepages, courseware, pre-print server material, patents and institutional intranets.

TechXtra

TechXtra concentrates on engineering, mathematics and computing. It gives you industry news, job announcements, technical reports, technical data, full text eprints, teaching and learning resources along with articles and relevant website information.

Just like general web search, searching the Invisible Web is also about looking for the needle in the haystack. Only here, the haystack is much bigger. The Invisible Web is definitely not for the casual searcher. It is a deep but not dark because if you know what you are searching for, enlightenment is a few keywords away.

Do you venture into the Invisible Web? Which is your preferred search tool?

Image credit: MarcelGermain

  1. deeznuts
    May 4, 2016 at 5:39 pm

    need one that lets me get on face book and past fire walls

  2. Rob
    December 8, 2015 at 1:28 am

    I use the following sites:

    CLuuz.com, search.carrot2.org, qwant, addictomatic.com, arXiv.org, and others. But these should get you started. You can use standard Google dorks here too.

    • Saikat Basu
      December 8, 2015 at 8:36 am

      Thanks for those links. Most of them are clustering engines, but useful nonetheless.

  3. dan
    November 22, 2015 at 6:55 pm

    infomine are offline :(

  4. John Ellerd
    November 12, 2015 at 12:47 am

    I keep getting a message of invalid for my phone number????????????????????????

  5. Gilbert Midonnet
    May 14, 2015 at 2:42 pm

    There are a lot of sites mentioned here that are now closed.

    • Saikat
      May 14, 2015 at 3:03 pm

      This article is five years old for God's sake! :)

  6. seekerofwisdom
    May 11, 2015 at 5:47 am

    I went to Infomine and it's offline permanently,just so you know to take it off the list.

  7. Joe rogan
    April 26, 2015 at 7:46 pm

    I am looking for all the best deep, invisible, etc websites on how to check databases free of courze.. In a (xls) database or what not...

  8. MARCO ALVARADO
    December 31, 2014 at 2:01 pm

    STILL TRYING TO KNOW ABOUT THIS

  9. MOUHAMADOU BASSIROU POUYE
    December 31, 2014 at 5:29 am

    I think it's useful and interessting to get to know abouit those research websites , because there is something that help us to get more choises and more alternatives right?
    THANX U

  10. Jbloggs
    June 10, 2010 at 1:50 pm

    Crap does no more a deep search than google, in fact worse than google or yahoo. waste of time.

  11. Roddy MacLeod
    May 26, 2010 at 12:21 pm

    Latest: 10 interesting websites #8, as selected by Roddy MacLeod http://roddymacleod.wordpress....

  12. joel gratulas
    May 18, 2010 at 7:10 pm

    there's no way size of the indexed web is 167 terabytes. i solely have a almost one terabyte of data in my computer. if you have a look at it, the article that wikipedia referenced is 13 years old - it's date is 1997. besides the "popular" search engines are much more powerful now. imho, no matter how hard you try with these "deep web" search engines, you wouldn't be able to come up with any web page that is not indexed by yahoo or google. maybe there are non-indexed pages on the web indeed, but the only reason is probably information security. and they're supposed to be non-indexed, even by these "deep web" engines.

  13. Mike
    May 17, 2010 at 6:36 pm

    I have used DEVONthink and DEVONagent with some success (http://www.devon-technologies.com/), but I have to admit I am not really sure how is completes it's searches. It is billed as a professional search and organization tool for journalists and researchers and I have been quite happy with it. (NOTE: Mac only)

  14. Yohan Setiawan
    May 15, 2010 at 6:00 pm

    wow this is very interesting...
    thanks man..!

  15. The Public
    May 13, 2010 at 2:00 am

    Maybe we can find Obama's birth certificate... lol

    • Aibek
      May 13, 2010 at 3:12 am

      :-)

  16. Anonymous
    May 12, 2010 at 4:17 pm

    That is without doubt one of the best things i have seen in my Life, Peothry in Motion, I love it, Keep up the good work and Thanks for sharing !

  17. CJ
    May 3, 2010 at 2:06 am

    I had no clue this was even out there! Great Info! Good looking out!

  18. Joe Gumshoe
    May 2, 2010 at 12:37 am

    I suspect the numbers earlier in the article might be off a bit. I can't say for sure of course, but I suspect the open web is far more than 167 TB. If you consider the video content I would bet that YouTube alone exceeds that.

    I do definitely appreciate the article, and I've found some sources for searching that I knew nothing about! We all get lulled to thinking that Google is everthing, but not when the dark net is involved.

  19. Leplan
    April 21, 2010 at 7:50 am

    Nice post. How about the Real Time Social Search Engines like Topsy Tweet Search and Buzzom People search?

  20. Dr.Varghese
    April 11, 2010 at 7:12 am

    Thanks for the information
    Keep up the Good work

  21. Rodddy MacLeod
    April 4, 2010 at 9:31 pm

    JournalTOCs is now the biggest, free and searchable collection of scholarly journal Tables of Contents (TOCs) http://www.journaltocs.hw.ac.u...

  22. Syne
    April 2, 2010 at 9:27 am

    We don't know what we already know. Networking is essential in key words, and intuition. So we can rearange the natural fractal ordination basicaly there is in this world our Universe.
    Let there be a insite contact with God- inner self to make the right ordination and to see; chaos is just a moment that only we don't see the ordination wich is always and already there !

  23. John Cook
    March 28, 2010 at 11:11 pm

    Some time ago, for a friend, I produced an exceptionally realistic drawing of a stack of three future DVX's titled "The New York Library" and each was proclaimed to be 2000TB. While no intention to mock reality was intended, it is a bit shocking to see that other than the over capacity DVD's this could be considered current and accurate.

    I for one would love to see ALL the available information, but categorized into sections. Start with science fact, then literature that is considered accurate. . . and somewhere near the end would be current religious beleifs.

  24. Anvita
    March 27, 2010 at 8:43 pm

    This will surely prove very useful.Thanks..

  25. Omnivorous
    March 23, 2010 at 4:11 am

    Test these against a known search. A search that I do almost daily turns up 4,000 results on Google.
    Infomine: Your search could not be processed due to an error.
    WWW Virtual Library: no results

  26. Rodddy MacLeod
    March 23, 2010 at 2:37 am

    Also of possible interest is: 10 websites to help you keep up-to-date with scholarly journal contents http://hwlibrary.wordpress.com/2009/10/21/10-websites-to-help-you-keep-up-to-date-with-scholarly-journal-contents/

  27. Rodddy MacLeod
    March 23, 2010 at 9:37 am

    Dale,

    See my post: Ten science search engines
    http://hwlibrary.wordpress.com...

  28. dale
    March 22, 2010 at 3:10 pm

    Thanks for the post! I wish there was a usable tech search site. I remember a time when you could type in a serial or model number into Poople and get diagrams, setup info . . sigh . . . now all you get is pure crap or $$$$$ sites.

    Here is one started by CMU sometime ago. Seemingly, it also is bent towards academics . . sigh . . where are the useful TECH search sites!?!?!?!

    Anyway, here's CMU's work on deep web.

    http://www.wolframalpha.com/

    • Saikat
      March 22, 2010 at 9:25 pm

      Hmmm...that's true. Let me work on that and see if I can come up with something.

    • pedro parkero
      April 1, 2010 at 12:15 am

      Yeah, I have the same wish as dale... just these past hour I wondered what are the chances of me finding a service manual to an MSI notebook I have to replace an internal DVD drive of, and so far, my search has gone dry...
      I guess it's too much to hope that it's in the Internet, huh... :-/

    • MikeS10
      April 11, 2010 at 8:37 pm

      Same as Dale - Tech search for diagrams etc related to m/s number etc for products would be ftw :)

  29. sheheryar
    March 22, 2010 at 3:01 am

    None of these web engines are useful to me, because I couldn't find an e-book or lecture of Design and analysis Of Algorithms By Annany Levitin.

  30. PaulT
    March 22, 2010 at 2:35 am

    Useful info. However, "Complete Planet" doesn't seem to have been updated since March 5th, 2004, and the date range search didn't work for me.

    • Saikat
      March 22, 2010 at 3:05 am

      The site is apparently being 're-engineered' and the new one will be reopened in late 2010.

  31. Rodddy MacLeod
    March 20, 2010 at 2:09 am

    Those 10 search engines are popular with information professionals.

    Here's another one that lets you can search the latest Table of Contents (TOCs) of 13,325 journals collected from 433 publishers (searching 348,158 TOC articles). http://www.journaltocs.hw.ac.uk/

  32. Tamal
    March 20, 2010 at 1:41 am

    Concise yet informative article. Great job. Keep it up.

  33. Rodddy MacLeod
    March 20, 2010 at 11:09 am

    Those 10 search engines are popular with information professionals.

    Here's another one that lets you can search the latest Table of Contents (TOCs) of 13,325 journals collected from 433 publishers (searching 348,158 TOC articles). http://www.journaltocs.hw.ac.u...

  34. Amuthan
    March 20, 2010 at 12:22 am

    You are doing simply fantastic job I love this site and hearty congrats a great job you are doing thanks for every thing

  35. vince820
    March 19, 2010 at 1:31 pm

    Great info, one quibble-it's mother "lode" not "load". Term comes from mining.

  36. dave tribbett
    March 19, 2010 at 7:21 am

    Great post.
    http://tastethecloud.com/conte... is a good article that adds some additional detail to the topic and a good set of links to the deep web search engines and other helpful sites.

  37. Pavel
    March 16, 2010 at 10:15 am

    It would be nice to see any documented research comparing the search results and their usefulness from the open web search engines and the invisible web search engines. In other words, how do I know that the results I'm getting on a certain subject with these 10 are better/more useful than what google/yahoo/bing/etc... would provide? I can and will test some of it myself, but as I said, it would be nice to see some good objective research...

    • Rodddy MacLeod
      March 20, 2010 at 2:07 am

      Excellent idea to actually do some research and compare these tools.

  38. Judy
    March 15, 2010 at 6:22 pm

    This is an excellent information find that I look forward to putting to use. I'm much more of a grandmother than a geek, but do use the internet to research things of interest to me & my family.
    I was surprised to find, when I checked it in print preview, that the first 2 pages of 7 for this article don't set up right to be printed off, which is what I was going to do (told you I'm not a Geek).
    Anyway, I'm pleased to have found this article & site.

    • Saikat
      March 16, 2010 at 6:25 am

      I tested out this post in Firefox and yes, it did not give a great preview of the print. But if you have Internet Explorer installed, then the Print Preview in that comes out okay.

      I am glad you liked the post. One of our basic purpose is to focus on technology that an everyman (or woman) can use. Do keep visiting.

  39. Benoneya
    March 15, 2010 at 4:16 pm

    Thank you, thank you:) I'm constantly looking for more research resources for the subjects I'm researching.

  40. Roberta
    March 15, 2010 at 3:55 pm

    Great article!!
    Thank You Very VERY Much.

  41. Rick
    March 15, 2010 at 2:15 pm

    Beautiful............thats like openning a few more libraries.
    Thanks

  42. Rodddy MacLeod
    March 15, 2010 at 9:19 am

    Good list of tools which will help undergraduates.

    Here's some more: 10 interesting websites #4 http://roddymacleod.wordpress.com/2010/03/11/10-interesting-websites-4-as-selected-by-roddy-macleod/

  43. David Rogers
    March 15, 2010 at 8:36 am

    Saikat, thanks. I´ve bookmarked most of these sites plus your article, in a Deep Search folder. One of the helpful items in your article directs me to a guide for learning how to search better. That´s where I´m going first. Later, when I have a topic I want to search in depth, I´ll be prepared for the hunt, thanks to you.

  44. Rodddy MacLeod
    March 15, 2010 at 6:19 pm

    Good list of tools which will help undergraduates.

    Here's some more: 10 interesting websites #4 http://roddymacleod.wordpress....

  45. Marcus Zillman
    March 15, 2010 at 6:25 am

    Resources for Deep Web Research
    htp://www.DeepWeb.us/

    • Saikat
      March 15, 2010 at 7:21 am

      Thanks for this.

  46. Herrnan Valdes
    March 15, 2010 at 3:52 am

    Thank you....will pass it on to my contacts.

  47. Saikat
    March 15, 2010 at 1:11 am

    Thanks, glad you liked it:)

    • Denissssee
      May 9, 2015 at 4:07 am

      Thanks for the help friend ly kisses

  48. vkvraju
    March 15, 2010 at 7:59 am

    Good info. Thanks,

Leave a Reply

Your email address will not be published. Required fields are marked *