Disaster recovery

What each mechanism covers

RAID is not a backup: it protects you from disk failures, not from deletions, ransomware or fires. The copy that leaves the machine is set up and tested following backups.md; this guide is what you do once something has already happened.

Before anything happens

Five minutes today, hours less anguish on the bad day:

1. An off-site backup configured and tested by restoring (backups.md).
2. A recent nas-config-*.tar.gz inside that off-site backup (not only on the NAS).
3. Alerts by email or webhook switched on (Control Panel → Alerts): a disk that fails or a backup that has been failing for a month has to tell you.
4. Written down somewhere outside the NAS: the pool names, the server name, the DDNS domain and the 2FA recovery codes.

A failed data disk

1. You will get an alert (Pool X is DEGRADED or a critical SMART reading).
2. Identify the disk in Storage → Health (serial number and status per disk).
3. Connect the new disk. If the machine supports hot swap you can do it without shutting down; if not, shut down (Control Panel → Shut down and restart), swap it and boot.
4. Storage → Health → Pool disks → "Replace" on the failed disk, and choose the spare. The operation runs as a background task: the resilver (ZFS) or the copy (Btrfs) can take hours, and the pool stays in use meanwhile.
5. Watch the progress in Storage (the "scan" line) until the pool is healthy again. **Do not remove another disk** during the process: this is the moment of least redundancy.

If you prefer the console, the equivalent is zpool replace <pool> <old-disk> /dev/sdX on ZFS and btrfs replace start <devid> /dev/sdX /mnt/<pool> on Btrfs.

Reinstalling the system while keeping the data

For when the system disk dies but the data disks are intact. The pools live on the data disks; the system is replaceable:

1. Install LGM-OS on a new system disk (install.sh or the ISO).
2. Restore the configuration: Control Panel → Configuration backup → "Restore from file…" with your latest nas-config-*.tar.gz, and restart the service. Users, groups and shared folders come back with their permissions; the session signing key is regenerated (everybody will have to sign in again).
3. Import the pools:
• ZFS: zpool import -f <pool> (or zpool import to list them).
• Btrfs: the restored mount units mount by label automatically; if not: mount /dev/disk/by-label/<pool> /mnt/<pool>.
4. Reapply the services from the UI (SMB/NFS/WebDAV are rewritten when you save).

Total loss of the machine

The NAS no longer exists: fire, flood, theft, a power surge that took the board and the disks with it, or ransomware that encrypted everything the NAS had mounted. The data is not on the machine, it is in your off-site backup, and the procedure consists of rebuilding the NAS around it.

Requirement: a recent, tested off-site backup. If you have never restored from it, today you find out whether it works. It is exactly the scenario backups.md tries to avoid.

Order of work

1. Hardware. It does not have to be identical to the old one. You need data disks with at least the capacity of what you are going to restore, and a disk for the system.

2. Install LGM-OS. Boot the ISO and answer the four questions (server name, network, console user and system disk). When it finishes, go to https://<ip>:5000 and create the administrator in the first-run wizard.

3. Restore the configuration. Get the nas-config-*.tar.gz from the off-site backup (it is among the copied files; if the backup is a USB disk, plug it in and take it from there) and upload it in Control Panel → Configuration backup → "Restore from file…". Then:

sudo systemctl restart nas-backend

Users, groups, shared folders with their permissions, services, network, scheduled tasks and alerts all come back. There is still no data and no pools: that is next.

4. Create the pools with the same names. Storage → "Create pool". Matching the old name matters: shared folders, exports and apps reference /mnt/<pool>/… paths, and with a different name you would have to redo them one by one.

5. Recreate the shared folders from the Control Panel (they are still defined after step 3: saving them is enough for them to be created on the new volume with their permissions).

6. Restore the data with the Backups app. The destinations and the tasks came back with the configuration in step 3, so the app already knows where your backup is. Two warnings before you start:

• If the destination is remote, the new NAS has a new SSH key: authorise it on the remote server just like the first time (step 3 of backups.md) or it will not even be able to read the backup.
• If the destination is a USB disk, plug it in and check that the destination shows as available.

Then, for each task: choose the version you want (usually the last good one) and the destination folder. Start with what cannot be replaced —documents and photos— and leave the media collection for last: that way you have the important things available in hours even if the rest takes days. If the task was encrypted, you need the encryption password; without it there is no restore.

If you prefer (or need) to do it by hand from the console, for example with a USB disk mounted on /mnt/usb-backup:

sudo rsync -aHAX --numeric-ids --info=progress2 \
     /mnt/usb-backup/<path-to-the-version>/documents/ /mnt/tank/documents/

Mind the trailing slashes: source/ copies the contents of that folder into the destination; without the slash it would copy the whole folder inside it.

If the remote destination was authorised with rrsync -wo (write only), that key cannot read: restore from the remote server itself or temporarily lift that restriction in its authorized_keys.

7. Reapply the services (SMB/NFS/WebDAV) by saving them from the UI and check from a PC that the shares show up and can be written to.

8. Docker apps. Reinstall them from the Package Centre and, before starting them, restore their data over /var/nas/apps/<app>/. Starting an app with an empty database and restoring it afterwards usually ends badly.

9. External access. The Let's Encrypt certificate is not restored: it is reissued on its own from Control Panel → Remote access as soon as the domain points at the new IP (check the DDNS and the port forwarding on the new router).

10. Get the backups running again. Check the destinations are available and that the tasks are active and scheduled. A restored NAS with no off-site backup is a half-recovered disaster: the next one catches you exactly like this one did.

11. Test the backup. Run a task by hand and restore a couple of files from the version it has just created. Close the loop the same day, not "when I get a minute".

How long it takes

Rough figures, so you can size your expectations and decide where to start:

The fact that restoring over the network is that slow is the reason to have also a local copy (or one you can go and fetch by car): the remote copy is your insurance against the fire, the local one is what gets you back to work the same day.

Restoring files from a snapshot

• ZFS: snapshots are visible at /mnt/<pool>/<dataset>/.zfs/snapshot/<name>/ — copy what you need without touching anything else. rollback from the UI reverts the whole dataset (and destroys later snapshots).
• Btrfs: "Restore" creates /mnt/<pool>/restaurado-<date> with the snapshot's contents; copy what you need and delete that folder (it is a subvolume: btrfs subvolume delete).

The configuration backup

Download it by hand from Control Panel → Configuration backup, or automate it: Control Panel → Scheduled tasks → type "backup" towards /mnt/<pool>/backups (weekly, for example). Keep that folder inside the off-site backup: it is the first thing you will need in the scenario above.

What the nas-config-*.tar.gz contains

• state/ — all the state in /var/nas/state: users and groups, shared folders with their per-user and per-group permissions, SMB, NFS, WebDAV, SSH, firewall, network, registered pools, scheduled tasks, alerts, notifications, App Store, SNMP, access portal, UPS and automatic IP blocking. This is what gets restored.
• etc/ — a copy of the generated files (smb.conf, exports, nftables, sshd, htpasswd) **for reference only**: restoring does not overwrite them, they are rewritten on their own when you reapply the services from the UI.

What it does NOT contain, on purpose

Only the machine's secrets and throwaway state; never configuration:

Any other state key goes into the backup by default, including those of future features: the table is the only exception and backend/tests/test_backup_state.py fails if somebody adds new state without deciding which side it falls on. It used to be the other way round —a list of twelve files to include— and shared folders were silently lost on every restore.

Restoring

Control Panel → Configuration backup → "Restore from file…". Only state/<key>.json entries in the format itself are accepted (any path leading outside the state directory is rejected) and the protected keys in the table are ignored even if they come inside the file. Afterwards: systemctl restart nas-backend and reapply the services from the UI.

⚠️ This backup does not include your data: that is what snapshots (inside the machine) and the off-site backup (outside it) are for.

What is still not covered

With the off-site backup configured, total disaster stops being a dead end. Even so, it is worth knowing where the limits are:

• Whatever happened since the last backup is lost. If you copy overnight and the machine burns at 20:00, the day's work goes. Raise the frequency of whatever you cannot redo.
• What was in no task is not there. A shared folder created after setting the backups up does not join on its own: review the sources of your tasks now and then.
• There is no image of the system disk (bare metal): LGM-OS is reinstalled from the ISO and the configuration is restored. It is quick, but it is not "clone and boot".
• If the backup sat next to the NAS, the disaster took it too. A copy on a USB disk permanently connected to the machine does not survive a fire, a theft or ransomware. That is why one of the copies has to be elsewhere.
• Sessions and 2FA: the signing key is regenerated and everybody signs in again; the TOTP codes are on each user's phone. Keep the recovery ones separately.
• The machine's keys and certificates are new: you have to re-authorise the NAS's SSH key at the backup destination and reissue the Let's Encrypt certificate.