Copy Methods

TAR over SSH

Single Thread

# tar cf - /source_directory | gzip -4 | ssh -p 22 -i /path/private_key user@host "cd /target_target_directory; gzip -c -d |tar xf - "

Multi Thread

# tar cf - /source_directory | pigz -p 8 -4 | ssh -p 22 -i /path/private_key user@host "cd /target_target_directory; gzip -c -d |tar xf - "

RSYNC

Handling sparse files

# rsync -avxS /source_directory -e "ssh -i $HOME/.ssh/somekey" user@host:/target_directory --progress

Normal copy with sma permissions

# rsync -avx /source_directory -e "ssh -i $HOME/.ssh/somekey" user@host:/target_directory --progress

With compression, traffic shaping 600K anda delete files different from source on target

# rsync -avx /source_directory -e "ssh -i $HOME/.ssh/somekey" user@host:/target_directory --bwlimit=600 --compress   --progress --delete

HTTP

Start a HTTP server with python then download everything with wget.