How to Monitor Websites that don’t have RSS Feeds

The problem with Mr Uptime is that if you have thousands, perhaps millions of people using the extension to constantly monitor a site to see if it’s still down or not, that page is going to STAY DOWN!! Because Mr Uptime is just going to constantly ping the domain over and over. So how on earth is the site ever going to come back up again if Mr Uptime is constantly pinging it?

It’s a good idea but poorly thought through.

Back in the day before my college newspaper had an RSS feed - I programmed a “scraper” in perl. This software basically grabs the webpage, and turns the headlines or text into an RSS feed. It works great but if they change their layout at all it no longer works. Plus there is that nasty issue of copyright. But, in an extreme case, you can turn a normal plain website into an RSS feed!

Certainly interesting for websites to be monitored in “real-time”. But if longer intervals (12/24 hrs) are enough, dedicated web services do the job just fine. I know since many years the following, :

http://changealarm.businessmatter.com/l

ChangeAlarm is faithful, regular, and really handy. They will notify by email any change in the url pages you will have notified. Easy, at no cost

Feed43 (Feed For Free) is a very useful online application that can churn out feeds from your favorite site that otherwise doesn’t provide RSS.

FeedYes is also a popular RSS feed service that gives users the same RSS functionality.

Thanks for posting about that FF extension Update Scanner. I’m going to try it out. I hate that there’s still interesting sites that still update, but are clueless about putting in a feed. I did the Bloglines/email thing to sub to many newsletters (eww), but Google Reader became stable at about the same time as Bloglines blew yet another gasket, and I got tired of their stupid plumber.

DWiner has a great point. There are many great websites out there that can change a page into an RSS feed. The one I’m thinking of is Page2RSS, which has worked wonders for me in the past. (http://page2rss.com/).

A problem I have had with Update Scanner is that it keeps track of all your websites in the TEMP directory. Whenever I ran a program like CCleaner, all the websites I’ve added to Update Scanner would disappear, and I needed to add them all over again.

I’ve been using WebSite-Watcher for many years now. It’s an excellent program, designed specifically to monitor wepages at certain time intervals.

[...] How to Monitor Websites that don?t have RSS Feeds | MakeUseOf.com - If you?re Firefox user then you can try Update Scanner. It?s an extension that lets you monitor webpages for changes. You have an option to choose how often the website should be checked for changes, and whether it should ignore the minor ones, such a [...]

I don’t think ChangeNotes works any more - it has not sent me information for quite a while and page takes forever to load. Thanks for the tip on ChangeDetection - will try that and see if it’s more reliable.

Keep up the good work…

[...] Read the full story… This entry was posted on Saturday, April 26th, 2008 at 7:49 am and is filed under le Chat Marchet. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. [...]

For webcomics you can try http://www.comic-nation.com
It only checks the images, so it can let you know when the comic updates.

http://www.changedetection.com is free and sends email every 24 hours. Worked well for me for years.

Yahoo pipes?

I’ve used http://www.dapper.net to create my own custom RSS feeds. So far just a few minor kinks I am working out

I was successful at making a quick and dirty RSS feed of one of my favorite bands The Vincent Black Shadow. Check it: http://www.dapper.net/services/TVBSRSS

Yahoo pipes and http://www.rssmix.com are a few other suggestions.

I say that if a site doesn’t offer rss I’m not going to follow it. I like using a good old fashion feed reader like Feedables

[...] good idea to write about it. So if you’re interested here are some free tools that can do the jobread more | digg story addthis_url = [...]

this good info …
i dont like yahoo pipes that much …
fan of google … hehe …

Check out http://www.openkapow.com. You download a RoboMaker application and visually create RSS feeds that you publish to their server. If the page changes, it’s easy to modify the robot you created and republish it. You can also create RSS feeds with images in them.

Pity, u r using linux

watchthatpage.com

I love and use UpdatePatrol: http://www.updatepatrol.com. It monitors both normal web pages and RSS feeds. It can highlight page changes and display them in a built in browser, or it can send you the page with changes highlighted by email. A real time saver.

The Firefox add-on Page Update Checker (PUC) is probably the most flexible and customizable web page checker, though not as “pretty” as Update Scanner.

I have used Watch That Page (http://watchthatpage.com/) for over a year now to check for software updates and other site updates. It emails out a daily digest with clips of changes. Highly recommended to try out. It’s also free, of course.

Personally, I always recommend using an online service versus a desktop app so you (as well as the application data) are not tied to your PC.

I just attended a conference where Kapow demonstrated their product, it was very impressive. I’ll be trying that one out at work, but it looks a little more complicated and full featured than what you need for this purpose. If you want to tie in these page changes to some other company service or do other SOA-ish integration, than it would be awesome.

[...] read more | digg story [...]

You could also simply use Google Alerts with a “site:” search term and something unique to the website - such as the header text on article postings.

[...] Update Scanner - Firefox extension that monitors webpages that don’t provide feed (via Makeuseof.com). [...]

Another way is to use Feed43.com service: it allows to parse any site and to create RSS-feeds

[...] How to Monitor Websites that don’t have RSS Feeds | MakeUseOf.com (tags: feeds tools extension Firefox) [...]

[...] How to Monitor Websites that don’t have RSS Feeds | MakeUseOf.com (tags: rss web monitoring firefox) [...]

[...] How to Monitor Websites that don’t have RSS Feeds | MakeUseOf.com If you’re a Firefox user then you can try Update Scanner. It’s an extension that lets you monitor webpages for changes. You have an option to choose how often the website should be checked for changes (tags: rss web monitoring firefox extension monitor website norss without feature) [...]

http://www.dapper.com could be an option

[...] read more | digg story [...]

How convenient, I was planning on programming this same tool however you guys just saved me the time and effort.

[...] found this via the helpful site, Make Use Of. Posted in Firefox Extensions | Leave a [...]

Before firing up an extra piece of software or service, check the source code of the site in question. You may just find out that it loads behind a frame. In that case, just open the frame to find the feed.

[...] How to Monitor Websites that don’t have RSS Feeds: Make Use Of mentions some ways to get notified about site updates when the site doesn’t offer RSS feeds. [...]

[...] How to Monitor Websites that don’t have RSS Feeds by Aibek @ [...]

[...] still don’t make these available, so what can you do? MakeUseOf.com provides an article on monitoring sites without RSS feeds that highlights a couple of tools you can use. Of course, you’ll need Firefox to use them, [...]