I recently needed to move a large number of files (millions) in a deep directory structure, into another similar directory structure, “merging” the contents of each directory and creating any missing directories. This task is easily (though slowly) performed on Windows with Control-C Control-V in Explorer, but I could find no obvious way to do it on Linux.
There is quite a bit of discussion about this on the web, including:
- Suggestions to do with with tar; this is a poor idea because it copies all the file data, taking an enormously long time.
- Suggestions to do it with “mv -r”… but as far as I can tell, mv does not have a -r option.
After a little thought I came up with the script below. I’d love to have a Linux/Bash guru out there point out how awful it and and send me something better!
A critical feature for me is that it does not overwrite files; if a source file name/path overlaps a destinate file, the source file is left alone, untouched. This can be changed easily to overwrite instead: remove the [[-f]] test.
$ cat ~/bin/move_files_merge.sh
#!/bin/bash
# Move files from dir $1 to dir $2,
# merging in to existing dirs
# Call it like so:
# move_files_merge.sh /FROM/directory /TO/directory
# Lousy error handling:
# Exit if called with missing params.
[ "A" == "A${1}" ] && exit 1
[ "A" == "A${2}" ] && exit 1
echo finding all the source directories
cd $1
find . -type d -not -empty | sort | uniq >$2/dirlist.txt
echo making all the destination directories
cd $2
wc -l dirlist.txt
xargs --no-run-if-empty -a dirlist.txt mkdir -p
rm dirlist.txt
echo Moving the files
cd $1
# There is surely a better way to do this:
find . -type f -printf "[[ -f '$2/%p' ]] || mv '%p' '$2/%h'\n" | bash
echo removing empty source dirs
find $1 -depth -type d -empty -delete
echo done.
