Kyle Cordes

I recently needed to move a large number of files (millions) in a deep directory structure, into another similar directory structure, “merging” the contents of each directory and creating any missing directories. This task is easily (though slowly) performed on Windows with Control-C Control-V in Explorer, but I could find no obvious way to do it on Linux.

There is quite a bit of discussion about this on the web, including:

• Suggestions to do with with tar; this is a poor idea because it copies all the file data, taking an enormously long time.
• Suggestions to do it with “mv -r”… but as far as I can tell, mv does not have a -r option.

After a little thought I came up with the script below. I’d love to have a Linux/Bash guru out there point out how awful it and and send me something better!

A critical feature for me is that it does not overwrite files; if a source file name/path overlaps a destinate file, the source file is left alone, untouched. This can be changed easily to overwrite instead: remove the [[-f]] test.

$ cat ~/bin/move_files_merge.sh
#!/bin/bash

# Move files from dir $1 to dir $2,
# merging in to existing dirs
# Call it like so:
# move_files_merge.sh /FROM/directory /TO/directory

# Lousy error handling:
# Exit if called with missing params.
[ "A" == "A${1}" ] && exit 1
[ "A" == "A${2}" ] && exit 1

echo finding all the source directories
cd $1
find . -type d -not -empty | sort | uniq >$2/dirlist.txt

echo making all the destination directories
cd $2
wc -l dirlist.txt
xargs --no-run-if-empty -a dirlist.txt mkdir -p
rm dirlist.txt

echo Moving the files
cd $1
# There is surely a better way to do this:
find . -type f -printf "[[ -f '$2/%p' ]] || mv '%p' '$2/%h'\n" | bash

echo removing empty source dirs
find $1 -depth -type d -empty -delete

echo done.

Tags: linux, programming

If you found this post useful, please link to it from your web site, mention it online, mention it to a colleague, or invite Kyle to speak at your next event.