Technology Explained

A Brief History of Search and How Google Came Out on Top

Bryan Clark 13-04-2015

After trying to love Bing How Bing Predicts Has Become So Good Time to quit your day job and start gambling via Bing Predictions? Not quite, but let's have a look at how Bing has become just so good at predicting the future. Read More , flirting with the enhanced privacy of DuckDuckGo 10 Useful DuckDuckGo Instant Answers That Save You Time & A Few Clicks Instant Answers is a collection of DuckDuckGo goodies that potentially make your online search life somewhat easier. Here are ten from a huge list you can try out on this alternative search engine. Read More , and (shutter) even trying to see if Yahoo was any better than the last time I used it (circa 2004), I still can’t find a better option for online search than the reigning champ, Google.


The gap is narrowing, but ultimately Google is still king due do the relevancy of the results, the speed at which sites are indexed and searches are executed, and their constant push to get even better.

After a half dozen (or more) unsuccessful attempts over the years to find a better alternative 13 Alternative Search Engines That Find What Google Can't Google Search still can't do everything. These 13 alternative search engines can take care of a few niche jobs for you. Read More , it led me to try and decipher what makes Google better than competitors in this space.

Here’s what I found.

A Short History of Search Engine Tech

First, I feel that it’s probably wise to look at some of the technology that drive search engines How Do Search Engines Work? To many people, Google IS the internet. It's arguably the most important invention since the Internet itself. And while search engines have changed a lot since, the underlying principles are still the same. Read More , and the companies that got us here.

The Creation of Archie (1990)



Archie is widely thought to be the first search engine. Although seemingly primitive now, Archie was a tool that indexed publically-accessible FTP sites in an attempt to allow users to find specific files. This was pre-world wide web, so you’d have to access the Archie servers using a local client or through Telnet What Is Telnet & What Are Its Uses? [MakeUseOf Explains] Telnet is one of those tech terms you may occasionally hear, but not in an ad or a feature laundry list of any product you may buy. That’s because it’s a protocol, or a language... Read More , which is a networking protocol that allows for the connection of two computers on the same network.

Archie spurred innovation in additional technologies that operated in a similar fashion, such as Veronica and Jughead, VLib, Gopher and others. These technologies were the pioneers of the search industry, even though they were quickly replaced by more advanced technologies.

World Wide Web Wanderer (1993)

Created in 1993, the World Wide Web Wanderer was the first bot created to index websites and automatically insert them into a database. The Wandex, as it came to be, started as a way to measure the size of the web by counting active servers worldwide in order to measure growth. Instead, it became the blueprint for modern web spiders, which are responsible for crawling the web in order to index its contents.

Excite (1993)



Excite was a project by six undergraduate students at Stanford University that attempted to use statistical analysis of relationships in common language in order to improve relevancy in indexing. Excite – then known as Architext – used journalists in order to write brief website reviews for improved usability when compared to more primitive link-based search engines and directories.

The Search Engine Boom (1994)

1994 was probably the single biggest year in the history of search engines. Within a 12-month span, we saw the launch of Infoseek, AltaVista, WebCrawler, Yahoo! and Lycos.

  • Infoseek: Allowed users to submit their webpages in real-time and indexed them automatically based on user-defined categories.
  • AltaVista: The first search engine to use unlimited bandwidth, allow natural language search queries, and the use of search modifiers in order to approve results.
  • WebCrawler: The first crawler that indexed entire pages as opposed to just titles or titles and descriptions.
  • Lycos: The first search engine to sort by relevance. Lycos also consistently had the largest database of stored sites, and at its peak (circa 1996), the service had crawled and indexed over 60 million websites.

Search engines during this time period (and several years later) made heavy use of meta tags that featured title, description and keyword elements added to a website by the webmaster. These tags, as well as keyword density within the text of a website, were the primary ranking factors that aided spiders in properly identifying and indexing content.

It also opened the door to web spam as early SEO experts quickly learned to manipulate this system by stuffing keywords into tags and content in an unnatural manner in an attempt to game the system.


Ask/Ask Jeeves (1997)


Ask – and their Ask Jeeves service – used human editors to try to list popular user search queries into listings of both partner and paid inclusion sites. The service did use automation in the form of a ranking system called DirectHit, which attempted to rank sites by popularity, but they found that this was too easy to manipulate and ceased use of the engine a few years later.

MSN/Bing (1998/2009)

MSN launched their own search engine to piggyback on the success of other Microsoft products and segue into online search. The search engine relied on Overture, Looksmart, and other paid inclusion directories in order to provide search results until Google proved the success of their backlink model.

This proof of concept by Google led MSN to adopt a similar strategy – although not nearly as good – in 2004.


In 2009, MSN rebranded as Bing, and improved the user interface, to include inline search (suggestions that appear as you type), content that appears on the search results page (as opposed to having to click a link to one of the search results), and a host of new features designed to make search easier and more accessible.

How Google Became Best (And Why It Still Is)

By the late 90s, search engines had become nearly unusable due to the relevancy (or lack thereof) of search results and the prevalence of web spam. Before its launch in 1998, and for half a decade after, competitors really had no answer to the mounting issues caused by web spam and search engine manipulation – which ultimately became known as SEO (search engine optimization).

Enter Google.


Google started as a project – known as BackRub – by Larry Page and Sergey Brin in 1996. The aim of this project was to revolutionize search by using links as the primary ranking metric in order to improve relevancy. The theory – known as citation notation – revolved around the idea that each site that linked to another site was essentially expressing their confidence in the content offered within its pages. These links served as votes, and the anchor text within the link served as a way to index sites based on what others were issuing these votes for.

The technology was exciting, and that led Brin and Page to attempt to sell it to competitors in the search world for a reported $750,000. Luckily, no one was interested. Back to the drawing board, Page and Brin continued to improve the technology and by 2000, Google was the largest search engine in the world.

How Google Stays on Top

Adaptability might be the best way to describe how Google remains the most-used search engine in the world. As marketers and search engine specialists learn to manipulate the system, Google changes it. When new technologies – such as social media – start to play an active role in our conversations, Google finds a way to integrate them into the algorithm to improve relevancy. When search habits change, Google changes with them.

Google is constantly watching, analyzing and evolving. By utilizing the massive amounts of data they collect from each of us using their products, they are able to constantly deliver a better, more relevant, and user-friendly product than any of their competitors.

What the Future of Search Looks Like


While Google is on top and there don’t seem to be any competitors on the horizon capable of unseating them, things change quickly on the web. Here is what Google – and others – are doing to improve the relevancy of our search results both now, and in the near future.

Understanding User Intent & Semantics

Search engines – including Google – used to deliver results based on how well queries matched up with webpages. Now, Bing, Google and Yahoo! are all attempting to improve their results based on understanding the intent behind them, known as semantic search What Semantic Markup Is & How It Will Change The Internet Forever [Technology Explained] Read More .

For example, if a user searches for “Best Seafood” – results in years past would rely on which website was deemed to be most authoritative for those keywords. Now, search engines are placing a greater emphasis on understanding what you are looking for specifically, so they can deliver the most optimized results. In this case, the user is most likely looking for the best seafood restaurants nearby, or recipes. Understanding that across trillions of possible search queries is where things get complicated.

Greater Emphasis on Location

The above example for a query about seafood is one of the primary reasons location is becoming so important. You probably aren’t looking for seafood restaurants in San Francisco if you live in Singapore (but if you are, just add “in San Francisco” to your query). Location-based search is changing everything as all major search engines are now utilizing your location in order to deliver customized results for your searches.

Improved Mobile Experience

Firefox Mobile

2015 was the first time in history that mobile traffic exceeded that of laptops and desktops, and it’s still a trend with rapid upward movement. In fact, most web traffic originating from smartphones is search-related. Because of the changing time we live in, it’s important to deliver a polished mobile experience to mobile users.

Google is doing just this by releasing a new mobile algorithm that hides sites that don’t display properly on mobile devices. This algorithmic change is only for mobile traffic, but it should enrich the experience for mobile users and lead the way for further mobile innovation.

Yahoo! is another major search engine with a focus on mobile. CEO Marissa Mayer (a former Google employee) has shifted the focus for Yahoo! in order to prioritize mobile as a primary means of regaining relevancy.

Knowledge Graph

Bing and Google are both largely invested in utilizing their massive databases of information in order to provide quick bites of information for specific keywords and phrases. In order to improve user experience, both are attempting to eliminate additional steps in which the user must take to find information, and instead relying on knowledge graph technology An In-Depth Look at Google's New Knowledge Graph Read More .

For example, movies, celebrity names, or location-related queries now feature information about the query at the top of the page. For quick searches, this may save you the additional steps of having to dig deeper through the results or visit multiple sites to find the information.

This has big implications on mobile, as well as voice search (such as Siri or Google Now) as most searches are quick attempts to retrieve infromation as opposed to exhaustive searches for research.


Finding New Ways to Improve Algorithms

Search engines are constantly at war with one another, as well as the unknowns that lurk just around the corner. In an attempt to be the best, each must deliver highly relevant results and quickly. As such, algorithms on all major search engines are constantly changing in an attempt to stay ahead of the curve, and find new metrics which may lead to better results.

One of the most exciting examples of this might be Knowledge-Based Trust score Can Google Use an Algorithm to Determine Truth? Google is researching whether its algorithm could include truth as a ranking factor. What does that mean for the web? Read More , based on a patent held by Google. While Google has outright denied any intent to use this patent in their search algorithm in the near future, the implications of it are huge. If you’ve ever been fed bad information after an online search, this new scoring system could change all of that by ranking sites based on results in Google’s Knowledge Vault (fact stores pulled from indexed content) and how well they match up.

Ultimately, trying to predict what search engine technology outside of the next year or two looks like is a supreme exercise in futility. That said, it should continue to get us one step closer to an age where we’re met with the answer to any question in a near-instant fashion.

Is there a better search engine than Google? What makes it better? What search-related technologies or changes are you most excited to see in the near future? Sound off in the comments below. I’d love to hear your take!

Image credit: Search engine via Shutterstock, Firefox Mobile by Johan Larsson via Flickr, Website code optimization via Shutterstock

Related topics: Google, Google Search, Microsoft Bing, Web Search.

Affiliate Disclosure: By buying the products we recommend, you help keep the site alive. Read more.

Whatsapp Pinterest

Leave a Reply

Your email address will not be published. Required fields are marked *

  1. Anonymous
    June 23, 2015 at 9:26 am

    ???? ?????? ??????????. ?????.

  2. Bart
    May 7, 2015 at 5:04 pm

    I was around on the web before google. I remember the day when I first heard of google. Sometime in 2001. My favorite search engine was Deja. I used to search the news groups which google bought. I was surprised to learn that Deja had a new name. Deja search had become google search although nothing had changed in the way the search engine functioned. It was the same but under new ownership. It remained my favorite until soon after google went public. Then I began to loose control of my search. The old search operators began to drop off unti it was no longer the old Deja I was so fond of. This was the beginning of google for me. Deja is an important element of the history of google missing in this article.

    • Bryan Clark
      May 8, 2015 at 2:24 am

      Throwback Thursday, huh? I used Deja a bit too, but it appears that we were in the minority. It wasn't ever that popular, which is why I didn't include it.

  3. Omar
    April 21, 2015 at 11:10 am

    There was a search engine which display what you are searching for in some kind of a cool video..but unfortunately i can't remember its name

    • Bryan Clark
      May 2, 2015 at 9:45 am

      That would have been cool to check out. If you think of it, please come back and let me know.

  4. Accessum Centrum
    April 17, 2015 at 10:08 am

    Thanks for this article.
    I don't know what I would do without Google and Wikipedia. There's another one that I find interesting. Wolfram Alpha calls itself a "Computational Knowledge Engine" rather than a search engine.
    Just for fun, type in your first name and click on the = sign. Or your first name, "and" and that of a friend. Then try surnames.

    • Bryan Clark
      May 2, 2015 at 9:45 am

      I use Wolfram Aplha all the time. However, it's not really going to give me the types of results that I'm looking for on some queries. It's more of a "we'll answer your question" engine (computational knowledge) rather than a "here are some things for you to read and learn about it on your own" search engine. Long story short, it's great for some things, but it's not really a "search" engine.

  5. John
    April 16, 2015 at 10:19 pm

    What about using sites such as
    As their advertising says "Search the best search engines, find the best results!" (includes Ask,Google,Bing,Yahoo)

    • Bryan Clark
      May 2, 2015 at 9:43 am

      I have yet to try it, but I will now. Thanks for the suggestion!

    • Bryan Clark
      May 2, 2015 at 9:47 am

      With a quick query, the results aren't all that promising.

      "how does a hard drive work"

      4 of the 10 first page results aren't really relevant to my query. They're hard drive related, but they don't answer the question. Relevancy is huge.

  6. Charrid
    April 16, 2015 at 6:40 am

    I liked the article.

    The reason I use Google to search is their timeline feature. If I want to find the latest news, say on Nokia, I can add that and set the search to the last 24 hours, I don't want to be given news of 5 months ago.

    • Bryan Clark
      April 16, 2015 at 3:35 pm

      Thanks Charrid! I make heavy use of this feature as well. It comes in quite handy when I'm researching time-specific stats for articles here at MUO and elsewhere.

  7. Leah
    April 15, 2015 at 5:51 pm

    Bing does provide credit, but only for 30 searches a day. You don't have to really do credible searches to get those credits. I don't. I prefer Google because they don't put their sites on top. Bing puts their MSN sites on top of all other results whether they are the best or not.

    • Bryan Clark
      April 16, 2015 at 1:30 am

      Search is very location specific now with most browsers making heavy usage of geo-targeting, in addition to the sites themselves actually tracking you. I say that because the following might not be true for you, but I actually didn't get the same results as you when testing this theory. Google actually showed more of their properties (Drive, YouTube, etc.), but the results were relevant to what I was looking for, so I really didn't mind.

  8. Alex
    April 15, 2015 at 7:44 am

    I think the part about understanding the users intent will become even more important in regards to these virtual assistants. As they develop we are being encouraged to speak to them more naturally but, whereas you want to be able to ask them a question, if you type the same ("how do I...?") into a search engine you'll get a bunch of forum questions you have to dig through . You're more likely to get the information quickly and concisely with a "how to..."

    • Bryan Clark
      April 16, 2015 at 1:28 am

      True. I think natural speech patterns are huge here, but I also think that the user has to recognize when to drop words in order to get different results. For example, "how do i swing a baseball bat" is going to show tutorials, forum posts, etc., while "swing a baseball bat" might show a crazy ex smashing your window.

      I made both of those examples up, but the meaning is there. Make sense?

  9. CoolHappyGuy
    April 14, 2015 at 1:09 pm

    You are quick to say what an also-ran Bing is but you haven't backed this up with specifics. I find Bing results to be pretty much comparable to identical Googles. I like its page layout a little better. Full disclosure: the REAL motivation behind using Bing is that they essentially pay me to do so via credits that turn into Amazon gift cards (other rewards are available). While I am a satisfied consumer of Gmail, YouTube, Maps (much better than Bing), Google Calendar, G+ (I use it more as an aggregator than socially -- for that Facebook still rules. Sigh.), etc; I Bing my generic searches.

    Another factor: the prevalence of "Google" as a verb. It's a form of free advertising. This is an area where Bing is at an huge competitive disadvantage. In the written world, you can't say you Binged the internet -- it looks to much like binge-d!

    • Bryan Clark
      April 16, 2015 at 1:26 am

      Here's a good example of what I mean. Try a search for "video sharing" on Google and Bing. Google has relevant results that are some of the top names in this niche, such as YouTube, Vimeo, Veoh, Vine, etc. whereas on Bing, the only video sharing site that I've actually heard of is Vimeo. That's not to say the others aren't good results, but they typically wouldn't be the big names that people are most likely looking for. While the results are relevant, they aren't relevant in the sense of understanding user intent and what they "probably" want to accomplish with the search.

  10. Phids
    April 13, 2015 at 7:48 pm

    I have long thought it a bit odd that no other search engine was able to compete with what Google did early on. I recall that when I first started using Google (probably in the late-90s/early 200s), I was impressed that it searched an entire web page - including text found near the bottom of it. While this may seem simplistic nowadays, the SERPs provided by other services did not do that (or at least it wasn't easy to find text that wasn't toward the top or more prominent in a page). I started using Google more frequently after that time because of this basic idea of relevancy, and this continues to this day.

    I should add that there is one other key reason why Google commands my control: integrated services, such as Drive, News, Calendar, Maps. I use those on a regular basis, so it's much easier to use one basic service to search with.

    • Bryan Clark
      April 14, 2015 at 12:24 am

      Ditto. Google is so much a part of my online life that it's getting a bit scary.

      BTW, did you know that Google attempted to sell to Excite for less than a million bucks? Man, would that have been the biggest bargain in the history of the world, or what?

  11. Greg
    April 13, 2015 at 3:25 pm

    I am really interested in hearing what you think about how Cortana for Windows 10 might change the market share for Google? Since Cortana will be backed into Spartan, I think people will become familiar with doing a real quick search to find info which is powered by Bing.

    Now, I still think for normal searches, people will just go to Google, but I could see this as Bing's big chance to capture market share.


    • Bryan Clark
      April 14, 2015 at 12:23 am

      I really thought the same thing about Bing, and while it did show improvement, Google just keeps innovating and dominating. I think until you get a company that has their type of corporate muscle (and proves they'll use it to improve), you aren't going to see any real threats come to fruition.

      By the way, Tina just wrote an amazing piece about Spartan... you should check it out.