How to find similar or duplicate files in Linux/Unix

[securitybreach]

I ran across this while searching for a command and thought I would share it:

Recently I have bought an USB hard-disk which is 2TB for $90 from US. And I have copied some English movies around 500GB from one of my friends Hard disk. Then when I am checking the content I found that some of the movies are repeated and that to with different names, in different folders etc. This made finding them difficult. So how to find all those files and deleted them if not required. There is one command in Linux called fdupes(File DUPlicatES) which is not installed by default. Lets see how we can use it.

 

In Redhat/Fedora/CentOS

#yum install fdupes

In Debain/Ubuntu

#apt-get install fdupes

Once installed you can use right away to find duplicates..

 

To find duplicates in a folder

#fdupes /path/to/folder

This will find all the duplicate files in that folder. What about if you want to find in sub folders too?

 

Use -r option to recursive search..

#fdupes -r /path/to/folder

but this will not show what is the size of that duped files? Then there is any option to find sizes too? Yes you can use -S to find sizes too..

#fdupes -S -r /path/to/folder

What about deleting them with a conformation so that you no need to go into every folder and delete them.

#fdupes -d -r /path/to/folder

Sample output

#fdupes -d -S -r /media/Movies/

 

[1] /media/Movies/ENGLISH MOVIES/The Pursuit of Happyness.avi

[2] /media/Movies/ENGLISH MOVIES/The.pursuit.of.happyness.(2006)/The Pursuit of Happyness.avi

 

Set 1 of 13, preserve files [1 - 2, all] (735221760 bytes each): 1

 

[+] /media/Movies/ENGLISH MOVIES/The Pursuit of Happyness.avi

[-] /media/Movies/ENGLISH MOVIES/The.pursuit.of.happyness.(2006)/The Pursuit of Happyness.avi

if you see i just deleted first files(+) and which is not deleted is shown in above output.

http://www.linuxnix....-linuxunix.html

 

We try to not post an entire article but this one was short.

[ichase]

Now THIS is definitely a command I find VERY useful. I have lot's of duplicate files. I move things around, forget to delete from source (I have always been bad about this) which of course eats up unnecessary space.

 

Thanks again Josh

[trigggl]

Thanks for this.

 

In Gentoo, to install, it's:

# emerge fdupes

 

Found a duplicate video or two and a duplicated ripped cd.

Edited June 1, 2013 by trigggl