Macs are great, with their neat UI and a Unix back-end. Sometimes you get the feeling you can do just about anything with them. Until one day you’re trying to do something simple and you realise what you need is just not available natively. In this case, I’m talking about wget.
In case you don’t know, wget is an incredibly useful command-line tool to download whole webpages, including media and as much link depth as you require. It’s quite powerful and sophisticated and it comes in very handy when you’re trying to make backups – especially since it can be worked into scripts. It doesn’t take a Unix guru to get hold of it and use it, either. It can be yours in a few steps, which are just fiddly enough to warrant writing out.
Get The Wget Binary
This is a cheat method which might not be around forever and may not work for everyone – but it’s quick. A very kind soul called Quentin Stafford-Fraser has made a binary of wget for Mac OS X Leopard. Download the zip file and unzip it. Copy the wget file to /usr/local/bin (and follow the README instructions for the other files also).
If this hasn’t worked for you for some reason, read on.
Get a C/C++ Compiler For Your Mac
If you don’t already have a C/C++ compiler for your Mac, you’re going to need one. The easiest way to get them is to install Xcode from the Mac Developer tools, which is available for free on your OS install disc or for Lion via the Mac App Store.
If none of these are an option, install the appropriate package according to the instructions at hpc.sourceforge.net.
Get XZ Utils For Mac
The wget package comes as a xz compressed file and it’s sad, but true, that this format isn’t supported by the MacOS Unzip utility. So, go grab a copy of XZ Utils for Mac so that you can unzip the wget package when you get it.
Get Wget Package Using HTTP or FTP
Unzip And Install Wget
Open up the Terminal application (Applications > Utilities > Terminal) and navigate to wherever the package was downloaded.
Unzip the file using XZ Utils and tar (replace the name of the file with whichever wget version you have downloaded):
xz -d wget-1.13.tar.xz tar -xf wget-1.13.tar
Change into the new wget directory and run these commands in order:
cd wget-1.13 ./configure make sudo make install
Some Ideas For Using Wget
Wget can be used in any instance where you might use DownThemAll to get pictures, music or video in bulk from a website. It can also be used to backup entire websites and perhaps mirror them for preservation.
If you’re still not sure what you might use Wget for, consider using Wget to back up your online photo directories or to grab MP3s from open folders online. Anywhere you know of where there’s a folder full of media or a page which links to or displays media can be downloaded with one command using Wget. Although it’s therefore possible to use Wget for entertainment you probably should have paid for, there are plenty of honourable uses for it.
For instance, if you work for a small company who are not so great with their backups, maybe you could use Wget in a script in order to back up your company website. When the inevitable day comes and the work server crashes, you’ll save the day by being able to pull up a recent backup. The same goes for your favourite community websites or anything run by amateurs: your kid’s footy club, the local drama group or whatever else you’re involved with.
Wget is especially good if everyone’s forgotten the password or if hackers changed your website password. Wget can grab all the important stuff for you and you can start fresh.
Here’s a quick example of a Wget command for creating a mirror all of the HTML and media from a page.
wget -m //www.makeuseof.com/
It’s simple and effective, although I honestly don’t recommend trying this particular example at home!
Cool Scripts And Downloaders
If you’re into automation and downloading, here’s some more to read:
- How To Do An Automated Remote Backup Of Your WordPress Blog
- How To Run Linux Style Cron Jobs on Windows
- How To Use RSS Feeds To Download Torrents Automatically
- How To Find Unprotected Website Directories & Get “Interesting” Files
What do you use wget for? Let us know!