How can I list all pages that belong to one domain?

HKB February 8, 2011
Pinterest Stumbleupon Whatsapp

Is there a way to list out all the webpages stored or linked to on a domain of the site? For example listing out all the pages that belong to MakeUseOf?

  1. iPhoneTutorialVideos
    June 18, 2011 at 12:57 am

     You can use a Sitemap generator program to at least get it into XML format http://www.xml-sitemaps.com/crawl.html

    Then use Notepad ++ to easily remove all the rest of the junk.

    Hope that helps.

  2. Tom Zucker-Scharff
    February 9, 2011 at 4:38 pm

    If you want you can use BackStreetBrowser to download the website in question

    • Hozefa KB
      February 9, 2011 at 7:21 pm

      you mean this will download all webpages hosted on the website?
      thats even cooler.

      Thanks, will check it out.

      • Aibek
        February 20, 2011 at 7:25 am

        If you're going to try this method make sure to keep the link depth at most 3 and not do that for large websites OR it may take ages to download the website :-)

  3. Josh Fox
    February 9, 2011 at 12:12 pm

    That can get messy with active blogs, but you could use Google with a modifier. Searching for "site:example.com" without quotes and replacing example.com with the domain in question. This will bring up everything that Google has listed for the domain, but you may need to tell it to repeat the search including omitted results. Some pages can be repeated because of how some sites work due to archives, categories, tags, etc.

    • Hozefa KB
      February 9, 2011 at 3:33 pm

      Thank you so much!!
      Your Method worked! the site domain didn't have many webpages so i looked through all of them. I was trying to find the main one.

      in case you are wondering, the site is:
      http://www.tiny.cc/pye

    • Hozefa KB
      February 20, 2011 at 10:26 am

      congratz!!

  4. FIDELIS
    February 9, 2011 at 6:11 am

    Have you tried any Whois tools?

    • Hozefa KB
      February 9, 2011 at 11:32 am

      I tried Who.is but it only shows the number of links but does not list them.
      whois.net requires signup, so i didn't use that

Ads by Google