Install from USB

This guide installs the first server of a new installation, which Velrix calls a realm. When it is done you open the server in a browser and finish in the setup wizard.

Before you start

Each server needs

* Not a measured limit: our test installations run on 4 cores and a 20 GB disk, and the comfortable figures are our recommendation. The memory figures are the stated requirement.

• Plan three servers or more. With three, the installation keeps working when any one of them fails. One server works, but has nothing to fail over to. Two servers are not enough for high availability.
• Keep them on one network. The servers of a realm find each other on their shared network: one LAN, VLAN or VXLAN. Between them, UDP ports 4397, 4400 and 6081 must be open.
• Open UDP 4443 to your users for the fastest connection to the web client. Without it, browsers fall back to a WebSocket after five seconds.

You also need

• The installer image and its checksum, from your Velrix contact.
• One USB stick for the installer, 4 GB or larger: the image is about 2.1 GB. Writing the image erases it.
• Two more USB sticks for the realm key, formatted FAT, exFAT or ext4. They keep the key that signs every server of your realm. Installing a VM or a VPS with no USB port? You can download the key as an encrypted file instead (step 5).
• An SSH public key, if you want break-glass access to the server’s shell as root. It is optional; there are no SSH passwords.

1. Write the installer stick

Check the image against its checksum, then write it to the stick. On Linux:

sha256sum -c velrix-os-<version>.iso.sha256
sudo dd if=velrix-os-<version>.iso of=/dev/sdX bs=4M conv=fsync

Replace /dev/sdX with the stick, and check twice: dd erases whatever you point it at. Any tool that writes a disk image works too. The same image also boots from a CD, on UEFI and on older BIOS firmware.

2. Let the server trust the installer (Secure Boot)

The installer is signed with the Velrix Secure Boot key. Most servers only trust Microsoft’s keys out of the box, so they refuse it until you add ours once:

1. Check the certificate’s fingerprint. It must be exactly:

   93:36:B8:4C:7A:96:3D:DE:10:8F:67:5C:EA:34:E7:A8:24:7C:2A:95:5F:0E:F4:71:2A:7E:40:2E:01:F2:9F:02
   

2. Enrol VELRIX-SB.cer from the stick into the firmware’s Secure Boot database (db): in the firmware setup (“enrol db from file”), or through the server’s management controller.

Fewer firmware visits: turn Secure Boot off for the install instead, and skip this step. Each installed server signs its own boot loader (step 7), so you visit its firmware once afterwards: enrol the server’s key and turn Secure Boot on, in the same visit. With a management controller (BMC), neither needs a visit: Redfish takes the certificate,

curl -k -u admin -X POST https://<bmc>/redfish/v1/Systems/<system>/SecureBoot/SecureBootDatabases/db/Certificates \
  -H 'Content-Type: application/json' \
  -d "{\"CertificateType\": \"PEM\", \"CertificateString\": \"$(cat VELRIX-SB.pem)\"}"

(openssl x509 -inform DER -in VELRIX-SB.cer -out VELRIX-SB.pem makes the PEM file.) Never turn Secure Boot off for good: it is what stops a changed boot loader.

3. Boot the installer

Boot the server from the stick. The menu offers:

• Install: the guided installer. Choose this.
• Install with safe graphics: for monitors that show nothing useful.
• Install text only: for serial and remote consoles.
• Unattended install: erases the only empty disk without asking. Leave it for scripted installs.

The installer shows itself on the server’s screen, and a line such as From another computer: open https://10.0.0.5/, check that the certificate’s SHA-256 fingerprint is …, and enter the code K7QM-2XFD. Use whichever you like: the console, or a browser anywhere on the network with that code. The browser warns once about the installer’s own certificate: compare the fingerprint, then accept it. The code changes after each use.

No network for the server yet? Use a cable. Plug a laptop straight into a free port of the server. With no DHCP server on the cable, both ends take a link-local address (169.254.x.x) by themselves within a minute, and the server’s screen shows its address and its name, velrix-install.local. Open https://velrix-install.local (or the address) on the laptop, compare the certificate’s fingerprint, and enter the code. Nothing needs installing on the laptop. The installer never hands out addresses, so the cable is just as safe on a live network. If two servers install side by side, the second one is called velrix-install-2.local: its screen says so.

4. Answer the installer

1. Language and keyboard. The installer then checks itself: every file on the stick against the checksums its verified boot menu carries, and the Velrix release against our signature. If anything differs, it stops.
2. Disk. Type the disk’s name to confirm. Everything on it is erased.
3. Network. Every port is listed with its link and speed.
   • Bonds: choose Add a bond to group ports, as active-backup (one port at a time, nothing to set on the switch), LACP (802.3ad: configure the switch ports as one LACP group first, or the bond has no link) or adaptive load balancing.
   • VLANs: choose Add a VLAN to put a VLAN (1 to 4094) on a port or a bond.
   • Addresses: give each connection IPv4, IPv6 or both, each automatic or fixed. On IPv6 alone, a server’s apps and modules reach IPv6 addresses, and IPv4-only services through your provider’s NAT64 if you turn on DNS64 for the network; its gateways offer no floating addresses or load balancers yet.
   • Management and mesh: mark the connection that carries the realm’s address and the traffic between servers.
   
   If you bond the port your browser reaches the installer through, its address moves to the bond: the page tells you the new address before the change is made.
4. Name. The server’s name.
5. Your SSH key (optional). A key for root over SSH, for when Velrix itself can’t be reached: paste it, or fetch keys from an https:// address such as https://github.com/<user>.keys (the server needs the network for that). Compare each key’s SHA256 fingerprint with ssh-keygen -lf <key>.pub on your own computer before you continue, or choose Skip.
   • Accepted: ssh-ed25519, ecdsa-sha2-nistp256 and -nistp384, and ssh-rsa of 3072 bits or more; at most 10 keys. A line with options or a command before the key is refused.
   • A fetch follows no redirect and takes https:// only; what it fetched is shown before it is used.
   • The keys become root’s authorized_keys on the installed server, owned by root. A new realm also puts them in its bundle, as before; the first bundle’s keys are added beside them.
6. Realm: Create a new realm. Give it a name and a colour. A new realm runs the whole product: apps, virtual machines, blueprints, volumes, networks, jobs, the registry, stacks, deployments, logs, metrics, Kubernetes and the rest. Minimal realm leaves all of that out, for special cases only.

5. Keep the realm key on two sticks

The installer now makes your realm key. Whoever holds it can make a server your whole realm trusts, so Velrix keeps it on no server at all. It goes on sticks, in your safe.

1. Plug in the first realm key stick (the installer sees it by itself) and choose a passphrase of 12 characters or more.
2. The installer writes the key there, encrypted with the passphrase, and reads it back to check it. The stick is relabelled VELRIX-REALMKEY.
3. Plug in the second stick, and write the same key to it.

You can skip the second stick only by confirming that you will copy the key yourself today. With one copy, a lost stick means making every server’s identity again.

No USB stick, as on a VM or a VPS? Choose Download to this computer (encrypted). Your browser saves the same encrypted file, velrix-realm-<realm>-issuer.age. Save it in two places, apart from the passphrase, then confirm I saved the file in two places. Only a browser that entered the code can download it. You need it again for an update that adds a module (Updates).

The server then forgets the key. It keeps only the public half, which is all it needs to check who belongs to the realm.

6. Install

Installing takes about half a minute on an SSD. The finish screen shows:

• The address of your installation, such as https://10.0.0.5/, with a QR code;
• The setup token, which proves in the next step that you are the one who installed it: six groups of four letters and digits, such as 7KQ2-…, with no letters that look like others. Case and dashes don’t matter when you type it;
• The server’s boot key fingerprint, for Secure Boot.

Write down the token. Remove the sticks and restart.

Put the realm key sticks in your safe, in two different places, and keep the passphrase apart from them. You need them once a year, and whenever you add a server.

7. Let the server boot with Secure Boot

Each installed server signs its own boot loader with a key sealed in its TPM. To boot with Secure Boot on, enrol that key once, the same way as in step 2 (in the firmware, or over Redfish): its certificate is EFI/velrix/VELRIX-HOST.cer on the server’s EFI partition, and its fingerprint is the one the finish screen showed. If you installed with Secure Boot off, this is the one visit: enrol the key, then turn Secure Boot on.

A server without a TPM has no boot key of its own: boot it with Secure Boot off.

8. Open the setup wizard

The server’s screen says Velrix is starting, then Velrix is ready with its address and the SHA-256 fingerprint of its certificate, usually within a minute of the login prompt.

Open the address. The browser warns once about the server’s own certificate: compare the fingerprint it shows with the one on the server’s screen, then accept it for this address. You now see Claim this instance. Continue with the setup wizard.

On a server with one public IP address, the realm and the apps you give hostnames share ports 80 and 443: see Public address and app hostnames.

Clients connect fastest over UDP port 4443 (WebTransport). If a firewall blocks it, they fall back to a slower connection after 5 seconds, so open it towards the server if you can.

Adding more servers

Every new server needs an identity signed with the realm key. You make it in a short ceremony:

1. Take a realm key stick from the safe and plug it into the first server.
2. In the Manager, open Realm key. It says when the stick is in.
3. Choose Add or re-issue an instance, type the new server’s name and the passphrase, and choose Sign with the realm key.
4. The server opens the key in memory only, signs, and forgets it. Choose Download the bundle: your browser saves velrix-instance-<name>.tar. Treat the file like a key.
5. Take the stick back to the safe.

Then install the new server from the installer stick. At Realm, choose Join an existing realm, then choose the bundle file (when you install from a browser), or plug in any USB stick that carries it and choose Look for a bundle on USB sticks. The server’s name follows the bundle (velrix-b for instance b) unless you typed another. Or let it boot diskless over the network. It joins the realm on its own, and the Realm key page says Instance b joined once its modules are connected.

From a join stick

For a site your realm can’t boot over the network, such as a branch office:

1. In the Manager, open Network boot, then Approvals → Make join stick. Give it a label (it becomes the new server’s), and choose Download the file: your browser saves velrix-join.json.
2. Copy the file to any USB stick (FAT, exFAT or ext4).
3. At the site, boot the server from the installer stick with the join stick plugged in too. The installer finds the file, checks itself, and connects to your realm at the address in the file only, checking the realm’s certificate against the pin in the file. Its screen shows the server’s host key.
4. Under Network boot → Approvals, compare that host key with the one on the server’s screen (ask whoever is there), and approve with the server’s bundle from the ceremony above. Leave the boot mode at Local disk.
5. The server installs itself to its only empty disk (or the Install disk you named), restarts from it and joins the realm. Take the sticks out.

The file works once, for a day by default. A lost stick adds nothing by itself: no server joins without your approval. The realm’s network boot address (port 8480) must be reachable from the site.

Every server’s identity is valid for a year. Operators get notices 60, 30, 14 and 7 days before one expires; the same ceremony, with Renew an instance’s identities, renews it.