MakeUseOf » duplicates

Find Duplicate & Similar Images On Your Hard Drive With SimilarImages [Windows]

Ann Smarty — Tue, 20 Sep 2011 01:30:43 +0000

The main reason why you would want to delete similar images is to save some space on your hard drive. Going through your image folder deleting extra images is too time consuming. Besides, many of the similar images can have different names which makes identifying them almost impossible.

Luckily, there’s a free tool to semi-automate the process.

SimilarImages is a desktop application that finds similar (or duplicate) images on your hard drive among different folders.

The tool has already been mentioned in our round-up of ways to find duplicate image files on a Windows PC but the tool definitely deserves a closer look.

Settings

The best thing about the tool is that it lets you configure the “similarity” threshold with enables you to configure which image pairs should be shown when comparing.

Image file pairs with a comparison value greater than the set threshold are ignored. Some recommended threshold settings include:

To find duplicates(fastest mode): 0 – 10
To scan through photos: about 12
To scan through folders containing scans: 50 – 60

You can also set the folder preferences:

Inclusive Search will only compare files located in the same folder with each other.
Exclusive Search means that only files not contained within the same folder will compared with each other.

Speed

If you have a lot of images and photos on your computer, the first check will take quite a while. However the great thing about the tool is that it keeps its own index of scanned images. Next check will run across its fast Graphics Library. This makes scanning much faster.

The program even keeps track of your not-a-duplicate decisions: if you enable Cache-System, SimilarImages will remember when you indicate that a pair is not a duplicate. When you later check the same folders again it would bring up the same false-positives.

You can access and clear the gallery from the tool “Cache” menu.

(Semi-) Automated Deletion

The tool also aims at making deleting duplicate files faster. Semi-automated deletion feature helps you choose which file to delete from a pair faster: when enabled, one file of a pair will be pre-selected depending on your rules.

For example, you can set to always pre-select a smaller image in a pair or the one with lower resolution or always the newer file. This definitely makes deleting extra images faster!

You can always set the tool to auto-delete a duplicate image in a pair (based on your above settings).

Working with Image Pairs

Once the application is done scanning the specified folders, it shows you the similar image pairs. Each image has a thumbnail, file size, dimensions, last-modified date and other properties. You can delete any of the files or replace one with the other.

The X-icon lets you delete the file.
The TwoFiles-icon lets you swap the file. (Swapping means in this context: The other file will be overwritten with this file, thus the contents of the first file will be kept but with the name of the second file.)
The Clear-icon lets you delete both files.
The Runner-icon lets you delete the file marked for semi-automated deletion.

Looks like a great tool to sort out your stored image files and free up some space. What are your thoughts?

Figure Out Folder Contents With Better Directory Analyzer

Danny Stieben — Mon, 19 Sep 2011 16:00:49 +0000

If you have a lot of files on your hard drive, you will probably at one point have wondered what’s in those folders you created a very long time ago. You might have some problems answering that question yourself, especially if the folder is full of millions of sub-folders and files.

Scanning through each sub-folder would take forever, and you could also skip over something that may be important to you. For cases like these, you need the right tool for the job. Thankfully, such a tool exists, and it packs some amazing features that will even make it fun to look through all those folders.

About Better Directory Analyzer

Better Directory Analyzer, which I will from now on call BDA, is a simple freeware program that scans through the contents of any selected folder and spits out more information about it than you thought was possible. All scanned files can be sorted in a handful of different ways, which helps you quickly find certain files that you may be looking for. So how is this helpful compared to Windows Search?

Although Windows XP‘s version of Windows Search actually packed more of a punch, Windows 7‘s version only lets you search by filename, file size, and date modified (without using hidden operators). Windows 7 users can find benefit the most by using BDA as there are many different parameters to search and select files.

Installation and Launch

To install, you’ll need to head over to this page and click on the download link. At the time of this article, there is only a version 1.0, which is fine. Once downloaded, double click on the file to start the installer. Go through all the steps, and you’re done. You should now have a nice shortcut for BDA on your desktop. When you first start, you’ll land in the Files tab. Here is where you can choose the folder you wish to scan for its contents.

Either drag and drop a folder into the empty box in the middle, or click on the “Select folder” button on the bottom and then choose the desired folder. It will then take a couple seconds (more if the folder is large) to scan for its contents, and when it completes you should see all the found files listed in the box in the middle.

Features

Great, you scanned for the files, but now what? Here is where the fun really begins. In the Files tab, you can also choose the minimum and maximum sizes for the files you would like to find, as well as a range for the date modified. If you move over to the File Types tab, you’ll find a list of all the different file types found in your selected folder.

A great feature of BDA is that it can visualize many of its results in pie charts, so you can adjust the number of specific file types that should be shown, choose whether it should count the number of files or the amount of space that each file type takes up, and then hit Visualize.

Under the Duplicate Files tab, you can easily spot duplicate files and delete extras where they are not needed. This tab is amazing if you know there are some duplicates and want to regain some wasted hard drive space. Here’s a simple tip: if you know that there are some files that are duplicated in two different folders, you can create a new folder and drag the other two into the new one. Then you can scan the new umbrella folder to include the files of the two other folders.

The Hidden Files tab is virtually the same as the tab for Duplicate Files, except that it shows hidden files. You can also select these and delete them as you wish. Finally, there is a Largest Files tab, where you can choose to view any number of the top files in terms of size. This helps find some clunkers that could be moved or deleted as needed. This tab also has a Visualize button.

The last tab in the program (besides Help) is the HTML Report tab. With this tab you can create a report (in HTML, I assume) that contains any of the selected bits of information over the scanned content of the selected folder. Use this tab if you need to transmit your results for whatever reason.

Conclusion

Better Directory Analyzer is a fantastic tool that I recently found and will be using extensively for a while to clean out some backup folders that I have sitting on my hard drive. In terms of tools that are aimed to be used for maintenance and similar tasks, this is one of the better programs that is actually a joy to use. There aren’t any complicated settings that need to be configured, so the experience is simply “install and go”. I definitely recommend this to anyone who could benefit from using it.

What techniques do you use to tame your files? What is your favorite feature about this program? Let us know in the comments!

Image Credit: Shutterstock

dupeGuru Finds Duplicate Files [Cross Platform]

Justin Pot — Thu, 08 Sep 2011 19:01:02 +0000

Quickly find duplicate files on your hard drive, so you can delete them and save precious storage space. You’ll be surprised how much of your storage is used to house the same files twice. There are many ways you can end up having duplicate files. Perhaps you copied a file instead of moving it, and failed to notice. Maybe you downloaded the same file twice without realizing it. Whatever the cause, dupeGuru can help you find duplicates so you can delete them.

A free, cross-platform tool, dupeGuru does only one thing – find duplicate files on your computer. As it turns out, it does this one thing very well. Capable of quickly scanning large volumes of files very quickly, dupeGuru scanned half a terabyte of files in a few minutes for me. If you’re running out of hard drive space CCleaner and CCEnhancer can help you free up space in Windows, and BleachBit cleans up Linux systems. If programs like that leave your drive too full however, finding and deleting duplicate files is a good idea.

Find & Remove Duplicates

Getting started with dupeGuru is simple – you need to add folders to scan.

Which folders you add is up to you – you can add just your Documents or the entirety of your hard drive. Be aware though that if it scans your system drive, it may alert you to duplicate files the system needs in order to run. Don’t delete anything you don’t understand.

The scan might take a while, but not too long – I scanned 500GB worth of data in just a few minutes. Once the scan is done you’ll have your results:

As you can see the system points out which files are duplicated and where. By default files are considered duplicates if their contents seem largely the same. As pictured you can see how alike files are by percentage. You can right-click copies to check them out or even delete them.

Would you prefer to find files with the exact same filename? This is possible too:

In the settings you change what a duplicate is. You can scan by the contents of the file, or just look for files with similar names.

Special Versions

Scanning by content or by file name might not give you the results you’re looking for. Music files, for example, can be encoded in different ways. An mp3 file may hold the same song as an mp4 file, for example, but look very different to the default version of dupeGuru.

Don’t panic! There is a special version of dupeGuru just for music files – dupeGuru ME. This version scans the metadata of your music files to find out when you have two copies of the same song.

This helped me find duplicate files I didn’t know about, and can probably help you too.

Are you a photographer? There is another similar tool for pictures. Find redundant copies of photos which are wasting space.

Download & Donate

Like what you see? Head over to the dupeGuru homepage to download dupeGuru. The music and the picture versions can be found there too.

If you use the program, and like it, you should seriously consider donating to the developer. He puts a lot of volunteer time into projects like this, and likes to eat food.

Conclusion

This isn’t the first time we’ve taught you how to find duplicate files. We’ve taught you 5 ways to find duplicate image files in Windows and showed you Auslogics’ duplicate file finder. I like dupeGuru however, because it works on Linux, Mac and Windows computers and is easy to use.

Do you like another tool better? Do you have any questions about dupeGuru? Let us know in the comments below.

Remove Duplicate Files & Directories with Duplicate Cleaner

Ryan Dube — Wed, 11 Aug 2010 19:31:06 +0000

One of the biggest problems with being addicted to the latest technology is that I’m constantly downloading new files – new installs for the latest apps, new add-ons for my browser or new themes for my blog. Then there are the thousands of images piled up from years of being a blogger.

After a while, I can’t remember whether or not I’ve already downloaded something. I must have three WordPress installs buried away in my hard drive. You can find files and directories that you think may be the same, but how do you know that they’re truly duplicates?

If you’re anything like me, rather than risk losing something important, you file away everything. Maybe you zip them all up and archive them. Maybe you back up everything to a big huge external hard drive for safekeeping. Ideally, it would be a lot better if you could chop down the number of files you are trying to manage by identifying true duplicates so that you can delete them.

We’ve covered a few tools at MUO for dealing with duplicate files, such as as the five utilities that Saikat covered, or the Auslogics finder that he also reviewed. An even more functional and effective tool that you can use to clean up duplicates is a freeware application is called Duplicate Cleaner.

Track Down Space-Wasting Duplicates

When it comes to identifying duplicates, Duplicate Cleaner takes all of the guess work out of the equation. It sorts through volumes of files for you, identifies every single duplicate file, and then returns a list with every last duplicate group.

The first step is to add all of the directories that you want to search through. In my own situation, it’s my laptop that’s the problem. I keep downloading lots of images and files to the Documents and Pictures folders. Just select each folder, one at a time, and click the blue arrow to add those directories into the “Search Paths” column. On the right side of the screen you can fine-tune the search criteria.

An alternative search is the “Audio Search” for audio files only – this is particularly useful if you have an insanely huge music file collection that’s a completely mess. Find duplicate music files within select directories, or you could even search your entire computer.

In my case, I’ve added the three folders where I always download files – downloads, pictures and documents. The next step is to let the software do its magic. It took Duplicate Cleaner just over 14 minutes to scan the entire three directories. I was astonished to learn that in two years I’ve accumulated over 35,000 files. The duplicate files represent 547.28 MB of wasted disk space.

The software splits up the resulting listings into two tabs – all searched files and the duplicate results.

Managing Your Duplicate Files

At first, I doubted that the software was really identifying duplicates. I had a lot of image files that had completely different file names, yet the software claimed they were duplicates. Luckily, all you have to do is right click on any file to open it. Sure enough, the images were exactly the same.

The software performs this same comparison with music files, video files and even executibles using what’s called the “MD5 Hash Algorithm”. The software comes with its own file preview utility, so even if you run it on a PC without any image app, you can view the images right inside Duplicate Cleaner.

Along with the file name, the results display the source path of the file, the file size, when it was last modified, and the group number. One clear giveaway that should tell you the software is working correctly is when the duplicate files always have identical file sizes.

The real question, once you’ve identified all of these problem files consuming your precious hard drive space, is what to do with them? You can select all of the duplicates that you want to delete, and when you click on “Remove Selected,” you will be provided with different options – delete the files, move them to a new folder, or, better yet, if you don’t really want to remove the duplicate file at its source location, you can create a “hard link” that is linked to the original file, and the duplicate is still deleted.

You can also fine tune exactly which duplicates are automatically chosen for deletion based on things like the age of the file, or in the case of audio – the one with the highest sample or bit rates.

Even if you don’t think you need such an application, you should give it a shot. I really didn’t think I had much wasted space, so I was shocked to find over 13,000 duplicate files in just three directories. Just imagine if you search your entire computer.

Do you have your own technique to prevent the duplicate file mess? How do you clean up your hard drive from this kind of wasted space? Share your own techniques in the comments section below.

How To Merge Duplicate Files & Folders With WinMerge

Ryan Dube — Thu, 03 Jun 2010 15:31:29 +0000

As I was browsing through old posts at MakeUseOf Answers the other day, I came across a question that gets asked often. How do you merge duplicate files or folders when you’re using a Windows PC? Linux users have simple tools they can use to quickly compare files, like the diff command for example.

Here at MakeUseOf, we’ve covered lots of apps that merge files, such as Mark’s article on how to merge PDF files, or Saikat’s article on how to merge video files. However, what if you just want to merge documents or folders filled with seemingly duplicate lists of files?

In Windows, the task isn’t quite as simple as it is in Linux, but the need does arise every now and then. Maybe you’ve had two common directories for a long time, and despite all efforts to keep them in sync, they no longer contain the same files. Or maybe you’ve been editing the same document in several locations, and you’re no longer sure which one is the most recent, and you don’t want to lose anything important in any one of the copies? The ability to compare seemingly duplicate files or folders can really come in handy sometimes.

An Open Source application that offers that “diff” functionality to Windows users is a free duplicate files utility called WinMerge. This program will accept three arguments – a file or folder to display on the left, a file or folder to display on the right, and a filter that advanced users can use to manipulate the comparison and merge.

Comparing Two Duplicate Folders

One of the most likely uses for this helpful application is to compare two files that you assume were synced, but maybe, for whatever reason, you’ve started to doubt that the two folders are truly duplicates. The WinMerge duplicate files utility can not only tell you whether the two folders are duplicates, but it can highlight the exact differences so that you can decide what you need to do to make the two folders true duplicates.

To portray how this works with folder contents, I’ve created two test folders that contain identical text and image files, except for two differences inside the folders. Folder number two contains an extra text file and an extra image file. As you can see above, I’ve selected folder1 and folder2 as the head to head comparison. Within seconds of clicking OK, the results were in.

As you can see, WinMerge immediately identified the two discrepancies. MyImage5.bmp and TextDocument4.txt are instantly identified as only existing in the folder on the right, Folder 2. The application even provides the file modification dates so that you can see whether there are even differences in when the files were modified. There are certainly cases where you may have a file with the same name, but one is updated while the other isn’t. This software will identify those differences as well.

Comparing Two Duplicate Files

To perform this experiment, I modified TextDocument1.txt and TextDocument2.txt in Folder2 so that they both contain identical text. Basically, I inserted the first three pages of Moby Dick into each file. Then, I inserted three unique sentences in file one, and one unique sentence in file two. The test is to see whether WinMerge can correctly isolate and identify all three differences between these two files.

Here, I’ve set document1 on the left and document2 on the right. In this case, no filters apply – I wanted to compare both files line for line and word for word.

As you can see above, the application successfully identified every single difference. The line containing the alteration is highlighted, and the alteration itself is further highlighted so that you can quickly scroll across the lines of text and see what the exact differences are. The nice thing about this is that if you’re working on a research paper and you’ve added entire paragraphs to the older copy, while you’re in this left window/right window mode, you can actually edit either document just like you would in any text editor.

Here, I’ve highlighted the unique line in document1 and pasted it into the same line in document2. You could do this for individual lines or entire paragraphs – edit either document however you see fit!

Performing A Merge

While manually going through the comparison results line by line and copying them over to the document of the latest version would certainly work, the software wouldn’t really be called WinMerge if there wasn’t some sort of automated approach to the process. Well, there certainly is. One of the top menu items is “Merge,” and the options under this menu item list everything you’ll need to merge any duplicate files or directories you may have.

The first nine options in this menu are actually what I would call “semi-automatic” merging. You can either use the menu options or the keyboard commands to step through each difference in the files, and then either copy the additional text to the right or the left document, depending which one you want to make your “master” document.

The last two options are actually the fully automated merge. Once you quickly scroll down through the differences in the files and you’re satisfied that you want to keep all of the unique content from one file and move it to the other, just select the file you want to merge to, and choose either “Copy All to Right” or “Copy All to Left.” These options automatically fly down through all differences and modify the master file so that it contains any additional content the other file may have. Once the process is through, your files or folders will once again be identical.

Have you ever had the need to compare or merge two duplicate files or folders? Have you ever tried the WinMerge duplicate files utility? Share your opinions and insight in the comments section below.

How to Easily Remove Duplicate Tracks on iTunes [Windows]

Jim Henderson — Tue, 06 Apr 2010 01:30:04 +0000

Today, I’m going to cover how to remove duplicate tracks from iTunes. During my recent Windows 7 rebuild, I made a number of strategic errors. Nothing devastating, unless you include the mess I seem to have got my once pristine iTunes library in. I’ve ended up with everything I need”¦ and more. I’m going to talk you through what you can do to remove duplicates in iTunes.

I’m a Windows guy, so some of your options will certainly differ if you’re one of those Apple types. Not that there’s anything wrong with that.

Remove Duplicates from iTunes – Before we start

We’ve talked about iTunes before. A lot! Relating to this particular subject, David Pierce explained how to use TidySongs to manage iTunes, and more broadly, Jackson produced a great eBook to help with all your iTunes needs.

How did this happen?

If only I knew. It’s a combination of recopying files to the machine after the rebuild, using the wonderful SharePod to rebuild the library, and some peculiar choices in the iTunes options.

Never mind. That’s ancient history, and there are many paths to this place. Doubtless, you took a different one.

Things to check

There are two things you want to make sure of before proceeding to remove duplicates from iTunes, and one general word of warning.

First, the warning. Please, please ensure you have backups in place in case any of this goes wrong. You are going to delete files, and you want to be sure you can get them back if nothing else.

If you are allowing iTunes to manage your files, you can just copy the entire iTunes folder (and all the subfolders) somewhere else. If you are managing the files yourself, take a look and be sure where you stand. Missing tracks is much worse than duplicate tracks.

There are two different ways that you can end up with duplicate tracks listed in iTunes. We ware here to remove actual duplicates. But first you need to check if you have duplicate items listed, but they are actually the same files. You can do this by sorting the tracks in name order (click on the Name column header), then right clicking one of the duplicates, and choosing Get Info.

Click on the Summary tab, and take a look at the file path after the Where: and then click on the Previous and Next buttons and compare the duplicates. Make sure the file names are at least a little bit different, or deleting one will delete both. In my case I have an Africa.mp3 and an Africa(1).mp3, for example.

The other thing to watch for is that the tracks are not near-duplicates. You know the stuff. You have the studio version, the live one, the unplugged one from the Nineties, the extended dance remix”¦ be sure you really want to get rid of those. Oh, lose the dance one. Go on.

Search and Eliminate

You probably already know you have some, and sorting your library by Title will certainly make it more apparent. If you have a small library, perhaps that’s all the help you need. It’s not going to work for me. I’ll still be working on it next year.

The first level of assistance is to ask iTunes to show you the duplicates. That’s easy. With your library still sorted by Title, click on File | Display Duplicates. This was when I started to get concerned. That’s a LOT of duplicate tracks.

If you want to reduce the work here a little, you can also ask iTunes to take a little more care with the selection. By default it’s only going to check the titles, but if you click on the Show All button to put things back to normal, and then go back through the process but this time hold the Shift key down while you click on File then you’ll get a Display Exact Duplicates option. For me, this made no difference at all but your mileage will vary depending on how you got the duplicates in the first place.

After that, the process consists of deleting the one (or more if necessary) duplicate files that you don’t want.

Getting help with the heavy lifting

I investigated making use of third-party products to take the back-breaking repetition out of the process. Dupe Eliminator isn’t a free product, but the trial version will let you make use of the comprehensive duplicate selection criteria. Unfortunately, it won’t then remove all the duplicates automatically. I can understand that though, because why else would you purchase the full version? I moved on.

You Apple folk also have a whole stack of Apple Script options, but I can’t get near those.

My next option was a free piece of software called Duplicate Music File Finder. This one works outside iTunes, so there’s a need to clean up the resulting dead links afterwards. Hold your breath, start the app, point it at your music files, and let it check them out.

Just before we go further, a reminder. I’m trying to help, but the risk is yours. You need to have backed up all your music before you start.

Click the Check All button if you are happy that the duplicates have been selected, and then click on Delete Checked Files to actually remove them. Click Yes to confirm.

The application behaved for me exactly as expected, and all the duplicate files were moved to the Recycle Bin.

The last step in the process is to tell iTunes to take a look at itself, and remove any items in the library which no longer exist. Unfortunately there is no simple way in the iTunes interface to do this, so we need to cheat, just a little.

Apple has provided a script to do just this particular trick, and to make use of it you just need to do these things in order:

Close iTunes if you have it open
Go to the web page, right-click on it, and save the page as a file, making sure you change the extension to .js. So for instance, in Firefox, right-click on the page, choose Save page As”¦ and change the file name from RemoveDeadTracks.txt to RemoveDeadTracks.js.
Open Windows Explorer and browse to the file you just saved
Double-click on the file.

The script will open iTunes, and then start running through and removing dead links. In my case it took around ten minutes and correctly removed nearly two thousand links.

Give yourself a pat on the back.

I’d love to know how you go, whether you have found any more comprehensive tools, or if you can fathom how I got in this mess. In the comments, please.

Trash Your Duplicate Files with Auslogics Duplicate File Finder

Saikat Basu — Sun, 06 Sep 2009 19:00:35 +0000

With human memory being what it is, we hardly realize that we stack up a pile of duplicate files on our hard disks. Downloading and saving files is no big deal. Especially with the kinds of bandwidth and hard disk capacities we enjoy these days. But gradually, over the days and months each byte accumulates like the proverbial drop in the ocean.

Most of us are pack rats of the digital age. Hoarding downloads, never to see them again. For instance, I had a habit of downloading e-books by the dozen – with each came a promise that I would read them all. Of course, I never got around to it. And soon, I came to realize that I had stacked up quite a few duplicate files.

It is only when we reach the brim of our hard disk capacity, we start to look for ways and means to shed some of the junk. An audit of our system will throw up at least a handful of files saved twice over. Image files and wallpapers spread across multiple folders. Duplicate songs, documents, outdated installation files, each contributing to sucking up space. Giving our systems regular spring cleanings not only frees up space, but also optimizes file organization, disk defragmentation and system scans.

Manually going through all our files with a fine toothcomb would be an absolute waste of time. Let’s find a smart software to do the job for us. Let’s get chummy with Auslogics Duplicate File Finder.

How does Auslogics Duplicate File Finder work?

The ‘identical’ file finder software relies on several reference points like the elementary matching by name, size and date. Apart from these (and more importantly), it also relies on matching by content. Matching by content of the file types is achieved using the MD5 algorithm. This algorithm is like a precise rule which can be used to find identical files by comparing their checksums. MD5 checksum is like a digital fingerprint of a file, unique and identifiable. Checksum match means that the files are identical.

Taking Auslogics Duplicate File Finder out for a hunt.

The freeware has a very simple intuitive interface. The process begins by selecting the drives and folders one wants to include in the search. The program also can scan all removable media like USB drives for duplicate content.

After specifying the where, in the next screen we have to specify the how. Three simple criteria cover all the options. With Matching Criteria, you can set it to search by file name, date and time of creation, file size and finally, match by content. Match by content is the most accurate as the first three can give differing results, but matching files by content improves accuracy. The only offshoot is that a search for checksum matches is considerably slower. The Advanced Criteria allows you to keep out files under a specific size. File Type further lets you narrow down the search to specific files.

Scan times depend on the search criteria selected. But my subjective opinion says that it’s quite fast. The results indicate the duplicate files and the potential space you can recover if you delete the duplicates. The duplicates are clustered together in groups and are marked out in alternating bands of color. File actions can be performed using a right click or from the Action menu. Actions like Select One in Each Group, Open With (the default file handler), and Explore Folder lets you better manage the clean-up. With a Delete, the duplicate files are moved to the Recycle Bin.

Note: Before deletion, use Action ““ Open to visually check the files as a precautionary step.

Rounding it all up

To list the pros – The 1.69MB download is free, uncomplicated and light on system resources (memory footprint isn’t more than a typical browser). Scanning is fast even with Match Content enabled.

To list the cons ““ The program could have done with more selections for file filters. The final action is limited to deletion; a move option from within the interface would have been a plus. The vanilla interface might put off users who are more hands on with their optimization tools.

As a typical no-cost, no-loss program, the Duplicate File Finder is worth a try. Auslogics Duplicate File Finder (ver. 1.5.2.55) runs on Windows (XP/2003/Vista/2008/7).

Which is your software of choice? Let us know how you purge the copies.

Image credit: where are the joneses

Plagiarism Checkers: 5 Free Websites To Catch The Copycats

Saikat Basu — Tue, 21 Jul 2009 20:01:36 +0000

“A lack of originality
Couldn’t focus on the day
so much for the radio
Everybody sounds the same
Everybody wears the same clothes now
And everybody plays the game”

That was from “˜Copycats’, a song by the Cranberries. Just swap the radio for the web and we could have an anthem for our times where content is free and content lifting is freedom. Copying is tame, plagiarism is the more serious (and demeaning) word and with the laws against copyright infringement in place, plagiarism as intentional fraud carries a mean look.

Content plagiarism on the web spreads as fast as the web itself. With blogging, which is usually written words, an original thought becomes the archetype . Duplicate content is the bane of webmasters and even bloggers. A webmaster can easily but unsuspectingly publish duplicate content on the fly (though a guy who knows his job won’t!). A blogger can see his work being scraped on another site. The bane ““ a lower page rank by search engines like Google which hates duplicate content.

Many tools exist which help us in sidestepping prickly copyright issues. The professional ones are of course potent, spiffy and dollar loving. The free ones may not be too much on looks but without a pinch they are adequate for the blogger world. Here at MakeUseOf, we have previously covered quite a few content tracking web services.

Here are five more for the armory.

Plagiarism Checker

A free online service which searches for originality (or lack of it) using a few phrases taken from different parts of the document. The phrases need not be contained within search operators like quotation marks as the plagiarism checker does it on its own through its code. Content checking is optional with either Google or Yahoo.

The two search engines though are word limited in the use of phrases ““ Google allows 32 and Yahoo’s word limit is 50. Plagiarism Checker thus limits the phrase within the word count. The word limitation restricts the scope of the content check. The matches open up in another webpage.

In a previous post we had looked at how Google Alerts could be used to keep track of your content. This service provides a button to the Google Alerts page for setting it up.

A webpage can also be submitted for a plagiarism check through a URL.

Article Checker

A very similar plagiarism detection tool which checks for duplicate content using either Google or Yahoo. Both the search engines can be used to compare results. Input of content is through a text area box or by giving the URL of a webpage. The checker functions by taking blocks of text and running it through the search engines. A Google Alert can be set up for the positive matches with a click.

Plagiarism Detect

This is a full-featured web service with a free and a paid option. The free option requires a signup and gives us an upload of Doc or Text files (or paste contents in a text area) and basic duplicate checking. The site makes a disclaimer that it does not store the uploaded content. Despite the basic checking service, the results are neatly displayed for us to sift through.

Quote Finder

This is a very simple content checking aid from Google Blogoscoped; a site which unofficially covers Google related news. The plagiarism checker is a simple text area for pasting the relevant text. The results are highlighted in yellow with the number of sources obtained from the web search. Clicking on the highlighted phrases takes us to the Google search results page. It is simple, uncomplicated and relies on the power of Google search.

DOC Cop

This free web service offers the option of a File Check and a Web Check. Using File Check we can scan and compare a maximum of eight Microsoft Word or AdobePDF files of up to 100,000 words each against one another. We can specify the average string length (i.e. the number of words) that must be compared. The report can be optionally obtained in HTML or Doc format through email.

Web Check is where we can put a webpage, a Doc file or a PDF file through a duplicity check on the internet. Web Check is limited to a maximum of 550 words. Again, the word length can be specified using the string length dropdown.

The web tool requires registration and access is through a guest ID. A real time status indicator shows the current status of the submission. According to the site, it takes about an hour to generate and send the report using email.

With the help of these free web tools, detecting duplicate content is not too tough a task. For a content writer, these useful tools serve another purpose ““ to determine where his ghost written articles have ended up”¦ or even where they have been re-used with due citation and credit. But mostly, it helps each writer check his work against the web to avoid unintentional duplication.

We have featured quite a few other tools here at Makeuseof.com ““

Copyscape : See Who is Stealing your Blog Content
Plagium: Online Plagiarism Tracker
The Plagiarism Checker: Check Papers For Plagiarism
CopyrightSpot: Search Web for Copies of Your Blog Content
FairShare: Find Out Who Is Stealing Your Content
CopyGator: Tells You Who Is Stealing Your Blog Content

The use of this rooster of article plagiarism checking apps should be enough to make us all tread on the side of caution and keep our creative spirits intact. If you write (or publish)”¦do you check? Let us know.

Image: swanksalot

5 Ways To Find Duplicate Image Files On Windows PC

Saikat Basu — Fri, 16 Jan 2009 20:01:52 +0000

If you are a quick fingered photo-snapper or a wallpaper junkie then the thousand of images on your hard drive could be a treasure hoard. But scattered among them could also be a few hundred duplicate images eating up space on your hard drive. Even worse, un-optimized raw images could be eating up chunks of bytes and can be officially called junk.

So, how do we go about covering up our laziness and cleaning up our image folders?

With some free software of course! Here’s a list of 5 software apps (in no relative order of merit) that can find duplicate image files.

Dup Detector

Dup Detector requires that all images be in one folder. After that it starts with its job by comparing pixel data for each image. It records it all in a data file and finds duplicate images by comparing the closeness of the pixel data. The software starts off by displaying a four tabbed interface ““ Method, Get Data, Find Dups and View Dups.

The ‘Method’ tab gives you the option to find duplicates in a single photo collection, between two collections or just compare a single image to all images in a collection. ‘Get Data’ enables you to open an existing data file (i.e. a list of images) or build a new data file. The ‘Find Dups’ tab is where you can play around with the matching criteria like percentage match, colour and luminance. The ‘View Dups’ is the results tab displaying the matched images. You can setup the software to do a manual delete, a semi-automatic delete or an automatic delete.

Dup Detector supports the following image file formats – jpg, bmp, png, tif, pcx, tga, wmf, emf, psp.

The installed application has a small footprint of 1.94 MB and can also be downloaded from this alternate site.

Supported OS: Windows XP and 2000 (Not tried on Vista)

DupliFinder

DupliFinder scans and checks image similarities by analyzing the MD5 sum (i.e. the digital fingerprint) of each file. The interface is stark and it lets you browse to your image folder or drag and drop it directly into the program. It lets you check folders or sub-folders for images which are identical and also for photos which are similar. Thus, you might get find related as well as duplicate photos in the results window. The preview helps you to choose which ones to delete. The scan speed is a bit sluggish especially for large image folders but the comparison results are close to the mark.

The 407 KB software runs as a standalone executable and requires Java 1.5.

Supported OS: Windows, Linux

DoubleKiller

As its name suggests, it searches for “˜double’ copies of all types of files employing any combination of name, size, modification date, or content. You can also employ wildcard searches and filter results using masks or attributes like file size range or files with certain properties like “˜read only’, “˜hidden’ etc.

The results are displayed at the end of the scan and here you can manually or automatically select the files to be deleted. The results can also be exported as a tab-delimited text file which can be imported into a spreadsheet application.

The 793 KB standalone program comes as a free version and also as a paid version with more sophisticated features.

Supported OS: Windows

SimilarImages

SimilarImages finds duplicate images on the hard drive using a “˜graphics library’ which checks the images for compositional similarities. The results depend on a configurable similarity threshold value which can be set by the user or left at the default mark. The program can handle a wide variety of image formats and searches for similarities using the Normal, Inclusive (only within the same source-directory), or Exclusive (Only outside the same source-directory) search modes. The program also has a feature to process only new files by a certain date.

The results window comes with two image panels and includes several actions like deleting one or both files and swapping files (replace one with the other). Using the “˜Semi-automated Deletion’ function one can also pre-select one file from the duplicates counting on the user-configurable rules.

Supported OS: Windows

VisiPics

I usually like to save the best for last and in my humble opinion it is this 1.9 MB software amongst the free bunch. VisiPics employs five image comparison filters to see how similar pairs of images on the hard drive are.

For instance, the software can mark out two identical image files but with different resolutions as duplicates, or the same picture but in different formats and even duplicates where only subtle compositional changes have taken place. Similarity checks can be set using a slider.

Detected duplicates are shown side by side as thumbnails and also in a larger preview window. The images can be flagged for deletion, they can be renamed and can also be moved to another location. The auto-select mode let you choose if you want to keep the higher resolution picture, uncompressed file types, smaller files or all of the above. The single window interface is easy on the eyes and the brain. It is also fast and as the software wiki says about the software – tested on 100.000 pictures, 15 GB archive, full results in 3 hours only.

Supported image formats include JPEG, GIF, PNG, BMP, PCX, TIFF, TGA and RAW

Supported OS: Windows

So if you are a photo buff, which software would be a must-install for you? Do you know of any other free tool which makes hunting down duplicate files free and easy? Let us know…