Jump to content

How to find similar or duplicate files in Linux/Unix


securitybreach
 Share

Recommended Posts

securitybreach

I ran across this while searching for a command and thought I would share it:

Recently I have bought an USB hard-disk which is 2TB for $90 from US. And I have copied some English movies around 500GB from one of my friends Hard disk. Then when I am checking the content I found that some of the movies are repeated and that to with different names, in different folders etc. This made finding them difficult. So how to find all those files and deleted them if not required. There is one command in Linux called fdupes(File DUPlicatES) which is not installed by default. Lets see how we can use it.

 

In Redhat/Fedora/CentOS

#yum install fdupes

In Debain/Ubuntu

#apt-get install fdupes

Once installed you can use right away to find duplicates..

 

To find duplicates in a folder

#fdupes /path/to/folder

This will find all the duplicate files in that folder. What about if you want to find in sub folders too?

 

Use -r option to recursive search..

#fdupes -r /path/to/folder

but this will not show what is the size of that duped files? Then there is any option to find sizes too? Yes you can use -S to find sizes too..

#fdupes -S -r /path/to/folder

What about deleting them with a conformation so that you no need to go into every folder and delete them.

#fdupes -d -r /path/to/folder

Sample output

#fdupes -d -S -r /media/Movies/

 

[1] /media/Movies/ENGLISH MOVIES/The Pursuit of Happyness.avi

[2] /media/Movies/ENGLISH MOVIES/The.pursuit.of.happyness.(2006)/The Pursuit of Happyness.avi

 

Set 1 of 13, preserve files [1 - 2, all] (735221760 bytes each): 1

 

[+] /media/Movies/ENGLISH MOVIES/The Pursuit of Happyness.avi

[-] /media/Movies/ENGLISH MOVIES/The.pursuit.of.happyness.(2006)/The Pursuit of Happyness.avi

if you see i just deleted first files(+) and which is not deleted is shown in above output.

http://www.linuxnix....-linuxunix.html

 

We try to not post an entire article but this one was short.

Link to comment
Share on other sites

Now THIS is definitely a command I find VERY useful. I have lot's of duplicate files. I move things around, forget to delete from source (I have always been bad about this) which of course eats up unnecessary space.

 

Thanks again Josh :thumbsup:

Link to comment
Share on other sites

Thanks for this.

 

In Gentoo, to install, it's:

# emerge fdupes

 

Found a duplicate video or two and a duplicated ripped cd.

Edited by trigggl
  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...