A dupes search results list is always initialized to be sorted by the Dupes column. Sorting the list by this column will bring the duplicate files that belong together into adjacent order. The results list of a dupes search features a special column "Dupes" where each group of dupes is referred to by a unique ID starting from number 1 for the first group detected. Tick the Invert option to find all singular/unique files quite useful when comparing two folders that should be in sync. Generally, SHA-1 is thought to be reliable enough for real world usage and hence it is also the factory default. (Click to see all)įor Content you can further define the method of comparison: The choices go from MD5 (fastest but least reliable) via SHA-1, SHA-256, SHA-512, to Byte-to-byte (slow but most reliable). The Dupes filter in Find Files on the Info Panel (F12). What counts as a "duplicate" can be defined in the Dupes tab: It's either match by Name, or Date (modified), or Size, or Content, or any combination of these. files that have one or more duplicates in the searched location(s). If the filter is active then only duplicate files are listed in the search results, i.e. XYplorer's Duplicate File Finder is implemented as filter Dupes in the Find Files tab. Those by content, can be useful for freeing up storage space. Finds duplicate files by name, date, sizeĪnd/or content in any location. Please let me know if I forgot any other tool which should have been in this list.Duplicate File Finder. Both the fastest and second fastest tool rely on SQLite databases and allow you to explore duplicates interactively, after they run. Though it remains one of the only tools which can do byte-by-byte comparison. It is interesting to see how the (arguably) best known solution fdupes was also the slowest. Compiling of the C/C++ tools tend to be a little fragile out of the main UNIX distros, which is a problem when working on a NAS.
#Fastest duplicate file finder portable
liten and fastdupes come close second and may be slightly more portable as they do not require to be compiled. Since I keep my pictures in a separate NAS storage, it is also useful how much each of this methods hinges on memory or CPU: Conclusionĭupd was the clear speed winner, also with an acceptable memory footprint. Here are the results, tested on a 7GB mixture of pictures, videos, symlinks, small files and recursively nested files: The folder was designed to contain 1195 duplicates in 325 clusters. Definitely too risky to run on the folder containing the pictures of my kids.įinally, I excluded those files that failed to find all correct duplicates in my test folder. I find this behaviour too aggressive for most users. I did not take into considerations those tools that do not have the option of a “dry run” or simple listing of duplicates, but instead attempt to delete or hardlink the duplicate files. Makefile would have needed fixing) or appear to have very limited support (less than 10 stars on github, slow or stagnant development, etc.). I have excluded from this comparison those tools which could not be readily installed on my macbook (eg. From there I downloaded and installed all free/open source command line tools.
Wikipedia has a pretty complete list of duplicate file finders. So I decided to compare the speed of most of them. And no easy way to find which one is best. I quickly realized that there are far too many tools which have been written for this task.
The obvious choice was to search for a command line tool. One could try with iPhoto or some other graphical interface, but the approach is simply too slow for a large library (over 100GB).
Now that I have reached the limits of my hard drive space, I figured the best thing to do was to remove some duplicate pictures. Too afraid of losing family memories, often I end up downloading pictures from my phone multiple times, “just in case”. I do try to save everything in one place and have backups, but it is difficult to keep track of it. Why a duplicate file finder?Īs a parent, my picture collection has mushroomed in size. Other python and perl based solution did also very well, often better than their C/C++ colleagues.
#Fastest duplicate file finder free
Amongst the free tools that were correct in identifying all duplicates in my test, dupd was the fastest. And because I could not choose which duplicate finder to try, I have decided to test them all. I needed to clean up duplicates photos from my personal library.
0 kommentar(er)
