Guides for operators

Backups

Scheduled, encrypted backups of the platform's state, volumes and virtual machine disks, where they are kept, and how to restore them.

Velrix backs up three things: the platform’s own state, your volumes, and your virtual machines’ disks. Every backup is encrypted with its owner’s own key before it leaves the server, and only what changed since the last one is sent and stored.

Everything here is in the Backups app. People and organisations back up their own machines and volumes there. Operators back up the platform in Manager › Platform backups, which also holds the backup targets.

What is backed up, and how consistent it is

WhatHowConsistency
The platform’s stateEvery module’s own data, all taken at the same moment from each module’s history.One consistent state of every module.
A volume on a btrfs serverA read-only snapshot, sent with btrfs send.As the snapshot’s moment.
A volume on a server without btrfsIts files, read as they are.Files still being written may be caught half-way.
A virtual machine with the guest agentIts file systems are frozen through the agent (fsfreeze on Linux) for the moment its disk is cloned, well under a second. Windows guests are frozen through VSS the same way, but restoring one has not yet been proven end to end: until it is, treat Windows backups as unverified.Application-consistent.
A virtual machine without the agentIts disk is cloned with the machine paused for a moment.Crash-consistent: as after a power cut.

Each restore point says which one it is.

Backup targets

A target is where the encrypted copies are kept. Add one in Manager › Platform backups › Targets:

  • S3-compatible object storage: Safespring, Elastx, or your own MinIO or Ceph. Enter the address (https://…), the bucket (which must exist), an optional folder in it, and an access key and secret key. The secret key is sealed at once and never shown again. Velrix talks only to the address you enter.
  • A folder on the server. Use this only with a separate disk mounted there: a backup on the same disk as the data is not a backup.

A target is tried before it is saved: a test object is written, read back and removed. If that fails, nothing is saved and you see why.

The target only ever sees encrypted objects with meaningless names. It learns their sizes and when they were written, nothing about what is in them or whose they are.

Schedules and retention

Open Backups › Schedule. An organisation’s schedule needs the Manage backups permission; the platform’s needs Operate the platform’s backups.

  • How often: from every hour to weekly, counted from an hour you choose (in UTC).
  • What: every machine, every volume, or both. The platform’s schedule covers every module.
  • What to keep: the newest points whatever their age, then the newest point of each of so many days, weeks, months and years. Points the rules no longer keep are deleted, with everything only they used.

Back up now runs everything at once, or one machine or volume from its own page.

Restoring

Open Backups › Restore points, pick what to restore and the moment.

  • Restore beside (the default) leaves the original alone:
    • a volume becomes a new volume, named after the original and the moment;
    • a machine is restored as a new machine: choose New machine, the same image as the original, and pick the restore point in its From a backup tab;
    • a module’s state is saved as a file for you to look at.
  • Restore in place replaces the original, after you confirm:
    • a volume must be detached from its app first;
    • a machine is stopped, its disk replaced, and started again;
    • a module’s state is replaced. The other modules keep theirs, so what they hold about it may no longer match. To take the whole platform back to one moment, use Restore the platform to this moment on a platform run in Activity.

Machines and volumes also have a Backups tab on their own page.

The recovery key

Each owner’s backups are encrypted with their own key. The installation keeps it sealed, but if the installation is lost, its keys are lost with it. Without the key, nobody can read the backups, Velrix included.

So download the recovery key from Backups › Recovery key and keep it somewhere safe, apart from the installation: in a password manager, or printed in a safe. Every download is recorded.

To read backups on a new installation, add the same target, then in Manager › Platform backups › Recovery key use Recover with the owner and the key file. The restore points appear again and can be restored as usual.

When something goes wrong

A backup that fails, a restore that fails, and a schedule with no good backup for two of its periods each become a notice to whoever set the schedule (operators for the platform), also by e-mail where the installation sends mail.

Space and time

  • Backups are deduplicated: unchanged data is never sent or stored twice, across runs and across your machines and volumes. A second backup of an unchanged 1.3 GB machine took 7 seconds in our tests and stored about 1 MB.
  • While a backup runs, the server needs free space for one copy of the largest machine or volume being backed up.
  • In our tests on one server (NVMe, local S3), a machine with a 1.3 GB disk took 33 seconds to back up the first time, with the guest frozen for under 0.1 seconds; restoring it as a new machine took about a minute until it answered.