Keep Your Remote Directory Up To Date With Rsync And Unison

If you’re reading this article, you’ve most likely used SSH to connect to a remote server and SCP (Secure copy protocol) to copy files between the computers. This works great if you just need to copy one file (or a directory) that you will not modify later. But, if you’re developing an app or a machine learning program that you want to run on a remote server, you will find yourself constantly copying files and directories. The easiest way is to copy the root directory of your project, but this means that each file in the directory will be copied every time, even if you change just one line in one of the files. If you have big files, this process will be quite inefficient. That’s why you should use Rsync or Unison.

Photo by Viktor Talashuk on Unsplash

Rsync

Rsync, quite literally remote sync, is a Unix tool for syncing files over the network. It comes built-in with many Linux systems, and it’s also ported to Windows. It uses SSH to transport files on demand. Rsync checks your source directory and compares it with the remote directory. When it detects that the modification time of a source file doesn’t match the modification time of the remote file, it initiates file copying. Files that are already up to date will not be copied to the remote machine.

Rsync is very simple to use. To sync a directory, run:

rsync -a mydir/ user@remote:mydir

This will copy the contents of local mydir to remote mydir . You need to have SSH access to the remote server to do this. The option -a means that copying is recursive and that symbolic links are preserved. I also recommend to always put trailing slash for the source directory. If there’s no slash after the source directory, then the source directory itself will copied into the destination directory. With the slash, the content of the source directory is copied. You can also put ./ for the source directory. In that case, the content of the current directory will be copied into the remote mydir . Remote destination is specified in the same way as for SCP: user@server:/absolute_path or user@server:relative_path_from_user_home .

You can also sync directories the other way around, from remote to local, however you can’t sync both ways at the same time:

rsync -a user@remote:mydir/ mydir

By default, Rsync doesn’t delete files. To enable deletion, use --delete flag. All files that exist in thedestination directory, but don’t exist in source directory will be deleted from the destination directory. Be cautious when using this flag, and actually, don’t use it if it’s not necessary.

Unison

Unison is made for syncing files simultaneously in both ways. You can think of Rsync as half-duplex and Unison as full-duplex. Unison doesn’t come preinstalled, but it’s available in almost every package manager on Linux, macOS and Windows. The list is here. On Ubuntu, you can install it with:

sudo apt install unison

Don’t forget to install it on both local and remote computer. The basic command to sync two directories using SSH protocol is:

unison source/ ssh://user@remote/destination

The slash after source directory is not necessary. To specify an absolute path for the remote destination, put two slashes after the remote hostname: ssh://user@remote//home/user/destination .

OK, let’s try it out.

Create a local mydir with files afile and bfile .

plavy@local:~$ ls mydir
afile  bfile

Next, make sure you have SSH access to the remote server and that the remote server has Unison installed. Now, we can run Unison locally.

plavy@local:~$ unison mydir ssh://plavy@remote/mydir

On the first run, you will get a prompt like this:

Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
 /home/plavy/mydir
 //remote//home/plavy/mydir
This can happen either
because this is the first time you have synchronized these roots, 
or because you have upgraded Unison to a new version with a different
archive format.

This happens because Unison didn’t find a state file for synchronization of these two directories.

Unison uses a state file to track what computer makes what changes to the files. It is somewhat similar to how Git and other version control systems work. Imagine two people committing to the same branch. Git needs to track changes and make sure that the two people are not trying to make a conflicting change to some file. Yes, Unison can also encounter conflicts when syncing files.

You can press return to dismiss the previous prompt. Unison will now show you what changes will be made. In our case, a directory will be copied from the local to the remote computer. You need to type y to confirm the update.

local          remote       
dir      ---->              [f] 

Proceed with propagating updates? [] y
Propagating updates

Now SSH to the remote server and check out mydir in your home directory.

plavy@local:~$ ssh remote
plavy@remote:~$ ls mydir
afile  bfile

Let’s create a new file on both the remote and local computer, and sync it.

plavy@remote:~$ touch mydir/cfile
---
plavy@local:~$ touch mydir/dfile 
plavy@local:~$ unison mydir ssh://remote/mydir

Make sure that you’re always running Unison on the same computer, i.e. on the local computer.

local          remote       
         <---- new file   cfile  [f] 
new file ---->            dfile  [f] 

Proceed with propagating updates? [] y
Propagating updates

Unison detected new files and will sync them simultaneously. Both local and remote directory now have all four files. That was easy, but what if we create a conflict? Let’s add different text to the same file and try to sync.

plavy@remote:~$ echo "important" > mydir/cfile
---
plavy@local:~$ echo "more important" > mydir/cfile 
plavy@local:~$ unison mydir ssh://remote/mydir

We need to resolve a conflict. Type either < or > to indicate direction of overwriting the file.

local          remote       
changed  <-?-> changed    cfile  [] >

I typed > , which means that the local file will take precedence. Finally, you can check the files to make sure the content is the same on both computers.

plavy@remote:~$ cat mydir/cfile
more important
---
plavy@local:~$ cat mydir/cfile 
more important

Conclusion

To conclude, both Rsync and Unison are good options for synchronizing directories on different computers. If you’ll be copying files only in one direction, use Rsync. If you’ll need to copy files both ways, use Unison. For example, Unison would be useful if you’re training an AI model on a remote server. First, you write the code locally, then copy the directory to the remote server with Unison, run the training, and copy the model data back to your computer with the same Unison command. No need to do manual copying with SCP. Also, Unison will always inform you about the changes that it’s going to apply, which is a good safety mechanism.