Goodbye Time Machine, Hello Rsync

Last week the screen on my laptop went black, and stayed black. It’s at that point your blood runs cold as you start to think “When did I last back up?”.

The good news is, only yesterday, and the even better news is: it’s in a format I can access (well mostly, we’ll come to that). The point of this isn’t to be smug, but to the thankful that only a few weeks ago, this would not have been the case.

Like many Mac users I’ve relied on Time Machine for my backups over the last few years. With little bother it will regularly and automatically back up your computer to another location, such as a NAS (as I do) or an external hard drive.

However, I was starting to get unhappy about a few things.

First, the size. Unless you limit it in a very none-obvious way the size of the backup will continue to grow until it’s used up all of the available space on the external drive. My laptop’s hard drive is only 128GB, and it’s not full, but over the last three years the backup file has ballooned to almost a terabyte.

The second is that the time machine backup is only really accessible to other Macs. If you don’t have a spare, or someone’s you can borrow for a while to extract your history, your only option is to suddenly buy another, at time not of your choosing. This could be expensive, both in cash and opportunity cost¹.

For those reasons I recently went to the age-old standby of rsync , which allows you to copy files to another location, and after the initial copy only copies the changes, making it quicker to run again, especially across a network.

The downside, compared to Time Machine, is that you can’t restore the whole system to a previous state, or any intermediate state. You can only copy the files as they were last synced to the remote location. In my case this isn’t a big loss, as I’ve never needed this feature. With hard drive space being cheap, I almost never delete anything.

The big benefit is that everything is visible in the file system, and not hidden in the Time Machine archive file.

Below is the final rsync command and options I settled on, stored in a script file that can easily be run manually, or via a cron job.

#!/bin/bash
rsync -aP --no-p --modify-window=1 --cvs-exclude --exclude '.DS_Store' --delete ~/Documents/ /Volumes/data/Backups/

The options mean:

• -aP: --archive (same as -rlptgoD (no -H,-A,-X), which in most cases are sensible choices) and -P which combines --partial (keep partially transferred files) and --progress (show progress)
• --no-p: --no-perms don’t preserve the files permissions. In this case there is only one user, and the archive location is trusted. This can sometimes cause issues if you want to look at the files on a borrowed computer with a different user, so I omit them. This should override the -p option included in -a above.
• --modify-window=1: Compare the modification times only to within 1 second. This was needed because the file system on the NAS wasn’t able to store timestamps more precisely, and rsync kept needlessly copying lots of files as the timestamps didn’t exactly match, even though nothing had changed.
• --cvs-exclude: excludes some files you normally don’t want to transfer, in my case it’s .git folders I want to ignore, as these are all synced to remotes, change often, and take a lot of time to check.
• --exclude '.DS_Store': Exclude this MacOS specific hidden file, that exists almost everywhere.
• --delete: Remove copies on the remote location if they no longer appear in the source. I rarely delete stuff, but when I do it’s normally larger files I don’t want to keep like ISO images, temporary copies of thinks like RAW photo files, all of which eat up a lot of space and don’t need preserving.

So, with this, everything is perfect, right? I can just use any spare computer to access the backed up data and carry on. Well, not quite. Even if almost all of my files are now easy to access in the case of catastrophic failure, I found there were two exceptions that I’d not paid attention to.

First; Encrypted Disk images. I prefer not encrypt the whole hard drive, as it makes recovery of files off it trickier if needed, and if you manually copy files across to another medium (like with Rsync) then they are not encrypted in their destination, unless you encrypt that location. Therefore my preferred solution is a portable encrypted container, in this case using a MacOS’s disk images, which you can copy wholesale between locations. If you need to access files on it, you mount it, just like an ISO image or a USB drive but requiring a password to decrypt, do you work, and then unmount it again.

This is a pain if you want to encrypt everything, but generally I don’t. Only sensitive documents, like bank statements, copies of important certificates etc. This means out of all my files, there are only a few gigabytes that I keep this way. While it might be a relatively small amount, it’s also some of the most important.

That’s why I was relieved to find HFS Explorer which can open encrypted disk images created in MacOS and that are formatted in the MacOS preferred HFS+ format. This is ok for recovery, but it can’t write to the image.

The best cross-platform replacement, that allows reading and writing is probably Veracrypt , which allows me to keep the same organising method I’ve been using all these years, but more reliably accessible on another operating system.

The second major item where I could loose something is my photo collection. Here I’ve used iPhoto and its successor Photos since 2005, meaning there is now more than a decade of photos stored in the library. The good news is that the .photolibrary ‘file’ is actually a directory, that does contain all the original photos that I have imported into it. This means no actual photos are lost, but any additional metadata or tags that I’ve added (such as organising photos into albums) is in a database that Photos uses internally, and isn’t easy to retrieve and re-associate with the relevant photos².

Note from the future: I did finally, in 2021, get round to exporting my photo collection .

Much to my relief the screen on my laptop has come back to life, but it appears to be a known issue around NVidia graphics cards on mid-2021 Retina MacBook Pros, and while currently intermittent, seems terminal. So while it still works for now, I’ll be backing up extremely regularly, and start searching for some kind of replacement.

This post ended up being the first in a loose series on Digital Independence . You can see the rest under the same tag .