Just want to share some rsync code snippets that I used to backup my data to either Dropbox or Jungle Disk on my Mac.

I have problems finding/Googling the proper flags to use for rsync for Dropbox and Jungle Disk when I first got started with rsync on the Internet, hopefully this blog post will be useful for someone.

As Dropbox is essentially a folder on your Mac, the rsync flags are pretty straight forward, however the same cannot be said for Jungle Disk as it is using WebDAV and storing the file over at Amazon S3 or Rackspace Cloud Files where file modification times will not be accurate.

• I am using rsync 3.0.8. You can compile rsync 3.0.8 on your Mac by following this tutorial. You just have to replace 3.0.7 with 3.0.8 in the commands.
• For Jungle Disk, I am using Rackspace Cloud Files instead of Amazon S3 because Rackspace does not charge for data transfer and hence it will be cheaper.
• I am doing an rsync from my remote host to my computer and any files not found on my remote host will be deleted on my computer.

/usr/local/bin/rsync -avz --delete --log-file='/path/to/Dropbox/logs/rsync/yourdomain.com.txt' --log-file-format='[%o] %f (%l bytes) (Last Modified: %M)' --exclude-from '/path/to/Dropbox/web/exclude_yourdomain.com.txt' -e ssh username@yourdomain.com:/home/username/public_html/ /path/to/Dropbox/web/yourdomain.com/

Rsync Flag Description

• -a aka –archive is a short hand for -r -l -p -t -g -o -D. Quick way of saying you want recursion and want to preserve almost everything
  • -r aka –recursive Tells rsync to copy directories recursively
  • -l aka –links When symlinks are encountered, recreate the symlink on the destination
  • -p aka –perms Causes the receiving rsync to set the destination permissions to be the same as the source permissions
  • -t aka –times Tells rsync to transfer modification times along with the files and update them on the remote system
  • -g aka –group Causes rsync to set the group of the destination file to be the same as the source file
  • -o aka –owner Causes rsync to set the owner of the destination file to be the same as the source file
  • -D aka –devices –specials Causes rsync to transfer character and block device files to the remote system to recreate these devices as well as special files such as named sockets and fifos
• -v aka –verbose Increases the amount of information you are given during the transfer
• -z aka –compress Compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted
• –delete Tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized
• –log-file Tells rsync to log what it is doing to a file
• –log-file-format Specify exactly what per-update logging is put into the file
• –exclude-from Specifies a file that contains exclude patterns (one per line)
• -e Specifies the remote shell to use

Layman’s Terms

• I am logging the rsync operation to the file /path/to/Dropbox/logs/rsync/yourdomain.com.txt with the log format [%o] %f (%l bytes) (Last Modified: %M) which translate to (as an example) 2011/07/14 08:00:00 [755] [recv] path/filename.txt (1024 bytes) (Last Modified: 2011/07/13-01:00:00).
• I am excluding all files and folders contained in this text file /path/to/Dropbox/web/exclude_yourdomain.com.txt. Each file or folder name you want to exclude should be on a new line in that text file.
• I am logging into yourdomain.com with the username username via SSH. You will be prompted to key in your password.
• Once my credentials are verified, I will be transferring all the files and folders in /home/username/public_html/ of the remote host to /path/to/Dropbox/web/yourdomain.com/ in my local computer.

/usr/local/bin/rsync --verbose --recursive --omit-dir-times --times --inplace --size-only --delete --log-file='/Volumes/webdav/web/logs/yourdomain.com.txt' --log-file-format='[%o] %f (%l bytes) (Last Modified: %M)' --exclude-from '/Volumes/webdav/web/excludes/yourdomain.com.txt' -e ssh username@yourdomain.com:/home/username/public_html/ '/Volumes/webdav/web/yourdomain.com/';

Rsync Flag Description

• –verbose Increases the amount of information you are given during the transfer
• –recursive Tells rsync to copy directories recursively
• –omit-dir-times Tells rsync to omit directories when it is preserving modification times
• –times Tells rsync to transfer modification times along with the files and update them on the remote system
• –inplace Changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file
• –size-only Modifies rsync’s “quick check” algorithm for finding files that need to be transferred, changing it from the default of transferring files with either a changed size or a changed last-modified time to just looking for files that have changed in size
• –delete Tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized
• –log-file Tells rsync to log what it is doing to a file
• –log-file-format Specify exactly what per-update logging is put into the file
• –exclude-from Specifies a file that contains exclude patterns (one per line)
• -e Specifies the remote shell to use

Layman’s Terms

• WebDAV folders are mounted to /Volumes/ and assuming you have mounted Jungle Disk WebDAV folder to be called webdav.
• I am logging the rsync operation to the file /Volumes/webdav/web/logs/yourdomain.com.txt with the log format [%o] %f (%l bytes) (Last Modified: %M) which translate to (as an example) 2011/07/14 08:00:00 [755] [recv] path/filename.txt (1024 bytes) (Last Modified: 2011/07/13-01:00:00).
• I am excluding all files and folders contained in this text file /Volumes/webdav/web/excludes/yourdomain.com.txt. Each file or folder name you want to exclude should be on a new line in that text file.
• I am logging into yourdomain.com with the username username via SSH. You will be prompted to key in your password.
• Once my credentials are verified, I will be transferring all the files and folders in /home/username/public_html/ of the remote host to /Volumes/webdav/web/yourdomain.com/ in my local computer.

Take a look at Rsync Manual Page for more detailed explanations.

I am up for discussions under the comments =)

I am hosting lesterchan.net on ex-FRRO shared/reseller server, now it can be considered a dedicated server that I am sharing together with Michael Feng. I have also a shared hosting account with a dedicated IP sponsored by Vodien Internet Solutions (which has excellent and fast customer service by the way). The FRRO server is using DirectAdmin while Vodien uses cPanel.

I have been Googling for days to find a way to replicate my files and databases from DirectAdmin (FRRO) To cPanel (Vodien) on a daily basis, but unfortunately I can’t find a way to do it. So I decided to think out of the box/back to basic and not do use DirectAdmin to cPanel but rather use rsync and mysqldump from one server to another.

I stumbled upon this article by HowtoForge entitled, Mirror Your Web Site With rsync. I followed the tutorial closely and it works perfectly, my files are mirrored from FRRO to Vodien everyday at midnight.

Within the same article, there are links to How To Set Up Database Replication In MySQL and How To Set Up A Load-Balanced MySQL Cluster. But unfortunately these tutorials requires root access to both servers which I do not have.

I thought of another idea which is to use mysqldump (at FRRO) to backup the database to a folder which will be included in the mirroring using rsync. After the rsync runs, the backup file should already be copied over to Vodien. Over at Vodien, I will use mysql to restore the database. This is quite a tiring process.

23:50 – Run mysqldump at FRRO (backup database)
00:00 – Run rsync at Vodien (mirror files)
00:30 – Run mysql at Vodien (restore database)

Notice that I give the cron jobs a buffer of 10 minutes for the backup database to happen before rsync and 30 minutes for the rsync to takes place before restoring the database.

Still not satisfied with the efficiency, I continued Googling for better solutions and I found an article by DevShed entitled, Backing Up and Restoring Your MySQL Database, in particular this paragraph:

Moving Data Directly Between Databases
How would you like to replicate your present database to a new location? When you are shifting web hosts or database servers, you can directly copy data to the new database without having to create a database backup on your machine and restoring the same on the new server. mysql allows you to connect to a remote database server to run sql commands. Using this feature, we can pipe the output from mysqldump and ask mysql to connect to the remote database server to populate the new database. Let’s say we want to recreate the Customers database on a new database server located at 202.32.12.32, we can run the following set of commands to replicate the present database at the new server

mysqldump -u sadmin -p pass21 Customers | mysql --host=202.32.12.32 -C Customers

And that is when I realized I can pipe the output of mysqldump of FRRO to the MYSQL server of Vodien. But before that can take place, you need to set the allowed host in Remote MYSQL in cPanel. In this case, I added the IP of FRRO as well as the hostname to the allowed host in Vodien’s cPanel.

I tested the MYSQL piping and it works perfectly! So now the cron jobs are reduced to just two.

00:00 – Run rsync at Vodien (mirror files)
00:10 – Run mysqldump at FRRO (pipe MYSQL database)

Here is my rsync cron job command that will run at midnight everyday:

/usr/bin/rsync -avz --delete -e "ssh -i /path/to/key/mirror-rsync-key" username@domain.com:/home/username/domain.com/public_html/ /home/username/public_html/

Here is my mysqldump cron job command that will run at 00:10 everyday:

/usr/bin/mysqldump --user="mysql_username" --password="mysql_password" --opt --all-databases | /usr/bin/mysql --host=192.168.1.1 --user="mysql_username" --password="mysql_password"

*UPDATE*

Uzyn suggested a more secured and faster way of “piping” the MYSQL database:

/usr/bin/mysqldump | gzip > /path/to/d.gz; ssh user@mirror "scp user@primary:/path/to/d.gz; mysql < gunzip -c d.gz"

What this command does is basically, gzip your database, ssh/scp it over to the backup server and gunzip it from there. It is quite similar to my rsync method mentioned previously.