Disks, spares and replicas

18 min · 10 sections

What you can do with the disks beyond creating a volume: speed it up, keep one spare waiting, send a live copy to another NAS and lend a disk over the network.

Hot spare

Storage → the volume's card → "Hot spare". ZFS only.

A volume with redundancy survives one disk dying, but it is left unprotected until someone swaps it. If that happens on a Friday night, the whole weekend sits one disk away from losing everything.

The spare is a disk that is plugged in, empty and waiting. When another one fails, ZFS puts it in its place by itself and the volume is protected again without anybody having to be standing there.

• When you add the first one, the panel turns on autoreplace, which is what makes the spare worth anything: without that property, ZFS leaves it sitting there and the volume degraded until somebody opens the panel.
• It adds no space. It is empty on purpose.
• When you add it, whatever it had inside is lost: it becomes ZFS's.
• When it steps in, the NAS tells you. That is the best and the worst news at once: a disk has died, but your data is already protected again. Swap the failed one to have a spare again.
• A spare that has already stepped in cannot be removed: that would not remove a spare, it would remove half of the volume's redundancy. Swap the failed disk first and the spare frees itself.

On Btrfs it does not exist: there a failing disk is replaced by hand from that same screen.

Read cache (SSD)

Storage → the volume's card → "Read cache". ZFS only.

An SSD in front of large disks speeds up repeated reads. It stores no data: what is inside is a copy of what is already on the disks, so if the SSD dies nothing is lost —it just gets read from the slow disk again.

There is no "add write log" (SLOG) on purpose: it only speeds up synchronous writes —a home NAS writes over SMB, which is not synchronous— and an unmirrored one that dies takes the pending writes with it. A button almost nobody needs and that can cost data is not a feature.

Replica to another NAS

Storage → Protection → Replication.

A backup gives you back your files. A replica gives you back the volume: the whole file system travels with all its snapshot history, so if this machine burns down, the one in the other house mounts the volume and carries on.

• Genuinely incremental: the first time everything travels and from then on only what has changed.
• On a schedule or by hand.
• Test the destination before saving. The typical failure —the SSH key not authorised on the other machine— cannot be discovered at three in the morning.
• The key is the same one as the backups use: whoever already authorised it has nothing to do.

What is needed on the other side

• A machine with ZFS (or Btrfs, depending on what you use here) and ssh.
• The public key of this NAS in ~/.ssh/authorized_keys of the user you give. The panel shows it so you can copy it.
• A destination that already exists: on ZFS, the parent dataset (tank2 to replicate into tank2/copy); on Btrfs, the folder.

What is worth knowing

• The first replica takes hours —everything travels— and has no time limit on purpose. The following ones take minutes.
• The destination is a copy: anything that diverged there is discarded on every send.
• If somebody deletes the replica on the other machine, the NAS notices and sends the whole volume again instead of failing forever.

Taking over: letting the second NAS start up

Replication sends the volume's snapshots to another NAS, whole and with their history. That solves half the problem. The other half is the one that decides whether it was worth anything: the day this machine dies, what has to be done for people to see their files again?

Until now the answer was a list of steps only the person who built the NAS knew: SSH into the second one, take the read-only flag off the volume, mount it, recreate the household folders with the same names, and change the server name. That is not a plan: it is a list of things that get done wrong at three in the morning.

How it works

The source machine writes down for the destination what it has to do: a small file, .lgm-relevo.json, inside the replicated volume itself, with its name, its SMB server name and its household folders including who gets into each one —the permission list, the NFS allowed machines and the rest of their settings—. It travels over the same SSH connection replication already uses. It is deliberately readable JSON: on the day it is needed, a person has to be able to read it with cat even if the panel will not start.

On the second NAS that plan shows up in Storage → Protection → Replication as "This machine can take over for NAS-Home", with a button. Pressing it:

1. takes the read-only flag off the replicated volume;
2. recreates its household folders with the same names and the same permissions — which is how the household finds their files again without changing anything. Accounts that do not exist on the second machine are named: create them with the same names and give them access again. And if a folder had NFS but the plan carried no list of allowed machines, it is recreated with NFS off and you are told: turning it on without a list would open it to the whole network;
3. and, only if you tick it, makes this machine advertise itself under the other one's network name.

A folder that already exists here is not overwritten: somebody may have created it earlier with its own permissions.

The rehearsal

Once a week the source machine checks — without writing anything — that the second one could take over today: that it can log in, that the volume exists there, that it has snapshots and that the plan is in place. If something that was ready stops being ready, it warns you. Same idea as the restore drill: prove it, do not promise it.

The rehearsal also fixes what it finds: if the plan is not there —or if it says anything different from what it would say today— it puts it back. That is what keeps the plan current: a folder created in May, or a permission granted last week, joins the takeover on the next rehearsal without anyone having to remember.

Why it is not automatic

With two machines and a single link, "the other one is not answering" and "the cable is cut" are indistinguishable from here. An automatic takeover would end up, sooner or later, with both NAS writing at once and two different versions of the same files. That is called split brain, and it is worse than being down, because it cannot be undone: there is no way to merge two histories that diverged.

So a person has to say "yes, the first one is dead". That is what the button is for, and why it asks for confirmation.

What taking over does not do

• It does not take the other machine's IP. That takes other pieces (VRRP, keepalived) and a different kind of network. What it can take is the SMB server name, which is how the household finds the NAS.
• It only works for ZFS replicas. On Btrfs the destination receives subvolumes, not a bootable clone of the volume.
• It does not fail back on its own. If the first NAS comes back to life, returning to the previous situation is separate work: you have to decide which version of the data counts, and that is a person's decision.

Coming from a Synology or a QNAP

Changing NAS is easy to say and very expensive to do: three terabytes to copy, twenty household folders to recreate with the same names, and a list of users nobody has written down anywhere. Most people do not switch because of that, not because of the price of the box.

The wizard in Control Panel → Bring over your old NAS data does the boring, exact part.

What it does

1. Looks at what is on the other machine without installing anything on it: it connects over SMB — which they all speak — with that NAS's admin account. It only reads: during the move, the old machine is still the one holding the good data, and nothing on this screen writes to or deletes from it.
2. Discovers the users. Neither DSM nor QuTS exposes its account list over SMB, but both keep personal folders under a share called homes, one per person: that list of folders is the list of users, and it is the only thing knowable from outside.
3. Copies folder by folder, creating the household folder here with the same name. If a name is not valid on this NAS — "My Photos" has a space — it proposes the one that is ("My-Photos") and says it changed it, instead of refusing.

The copy can be stopped and resumed: what already arrived is not downloaded again. And when it finishes, the files are counted on both sides, folder by folder, and it says whether anything is missing.

The old NAS's credentials are not stored

They live only while the move lasts, in a file that only the NAS's helper can read, and they are erased when it finishes. Keeping another machine's admin password for a move that lasts a weekend means leaving it there forever. They do not appear on the command line either, because during the hours the copy takes, anybody looking at the process list would see them.

What canNOT be brought over

This is said before you start, not in a footnote, because a migration that promises too much leaves somebody believing they have what they do not — and that gets discovered weeks later, when the old NAS has already been sold:

The order that hurts least

1. Bring one small folder first and check it: open three files.
2. Create the users it found, with temporary passwords, and tell them.
3. Bring the rest overnight, when the network is free.
4. Do not sell the old NAS until you have really restored something from this one. A month with both switched on costs little and avoids the one irreversible mistake of the move.

Encryption and its keys

Encrypting a volume was already possible at creation time: ZFS does it natively. What was missing is what comes afterwards, which is where encryption turns into a problem instead of a protection.

Storage → Protection → Health and encryption → Encryption and keys has three things:

Changing the password

If somebody who should no longer know it learned it — a technician, a former partner, a flatmate — until now the only way out was creating a new volume and copying three terabytes. Nobody does that, so in practice the password was never changed.

ZFS keeps a master key and wraps it with your password: changing it rewraps that key and does not re-encrypt a single byte. It takes the same time with one gigabyte as with twenty terabytes, so you can do it the day you need to.

Checking that you remember it

With the volume already unlocked there is no sign of whether you know the password: it is only needed at boot. If you wrote it down wrong, you find out months later, with the NAS down.

The Check button says so dry: it unlocks nothing, mounts nothing and stops nothing. Check it now and then; it takes a moment and avoids the one mistake that has no fix.

The key on a USB stick

A power cut at four in the morning leaves the encrypted volume locked, and with it there are no files, no backups and no cameras recording until somebody gets home and types the password. For many people that alone is reason enough not to encrypt, which is the worst possible outcome.

With the key on a USB stick, the NAS unlocks the volume by itself while that disk is plugged in. What is stored there is the same password, not a different one: if you lose the stick, the typed password still works.

And what that means, said plainly: whoever takes the NAS with the stick in it takes the key too. This protects against a stolen disk — or an old disk ending up in a skip with your photos on it — not against the whole machine being stolen. If the latter is your worry, keep the password typed and store the stick somewhere else.

The recovery sheet

The Recovery sheet button gives you a printable text with what to do if this panel will not start: how to import the volume and unlock it from any Linux with ZFS.

It does not carry the password, and that is not an oversight: the NAS does not store it anywhere, and that is exactly why the encryption is worth something. A sheet that carried it would be the link that breaks everything else. If nobody remembers the password, that volume's data cannot be recovered: not by the manufacturer, not by this panel, not by anybody.

External disks (USB)

Storage → Devices → External disks.

The USB stick or disk you plug in for a while: to bring over the photos from the old laptop, or to take some files to someone's house. It is not a NAS volume and it does not need to be; nothing you already have mounted is touched here.

Plug the disk in and it shows up in the list by itself, with its name, its size and what it holds. Three buttons:

• Open. The disk appears in Files as one more folder, under whatever name it carries, and can be read and written from the browser or from the phone. Only whoever administers the NAS sees it: a stick with the family photos is not shown whole to every account.
• Eject. Flushes to the disk whatever was still in memory and closes it. **Always do this before pulling the cable**: what you copy can take minutes to actually reach the disk, and pulling the cable early takes a half-written file with no warning at all. If something on the disk is in use, the panel refuses and says so — do not let being in a hurry win.
• Format. Leaves it empty as exFAT, the only thing Windows, macOS and Linux all read and write without installing anything. Everything on it is erased and there is no going back; the panel asks in those words and lets you name it (up to 11 letters).

Worth knowing:

• Disks from Windows (NTFS), FAT32, exFAT and Linux (ext2/3/4) can be read. A Mac disk with APFS or an unusual file system still shows up in the list, but cannot be opened and says why: format it if you want to use it here.
• A disk with several partitions is opened at the first one the NAS can read, and the panel says so. If you need the others, get them from the machine where they were made.
• A USB disk that is part of a NAS volume (an external enclosure holding a member disk) does not appear here: that is what the volumes screen is for, and taking it out of the set is a different thing that warns on its own.
• If you want to keep the disk inside the NAS —as part of a volume, with its snapshots and its RAID— this is not your screen: that is creating or expanding a volume.

Where each disk is heading

The Health screen says what SMART says: "fine" or "failing". That verdict arrives far too late, and it is worth knowing why: the threshold for each attribute is set by the manufacturer, and it is set where there is almost no margin left. A disk can have eighty reallocated sectors, hundreds of corrected reads, and still say "fine", because the threshold sits at several thousand. By the time it says "failing", something has usually already been lost.

What actually predicts a failure is not the number: it is the number rising.

Every six hours the NAS takes a snapshot of four counters per disk — reallocated sectors, pending sectors, uncorrectable sectors and reported uncorrected reads — and compares. If any has risen, it warns you even if the absolute value is tiny, and it states the rate: "three sectors in twenty days; at this rate it would reach 100 in about X days".

Disks are tracked by their serial number, not by /dev/sdX, which moves around on reboot: warning about the wrong disk is worse than not warning at all.

The cable is not the disk

There is a fifth counter, transmission errors (CRC), which rises because of a bad or badly seated SATA cable, not because of the disk. Almost everybody confuses the two, replaces a perfectly healthy disk and leaves the problem inside the NAS.

That is why it is shown separately and spelled out: turn the machine off, reseat that cable at both ends, and if it keeps rising, replace it. It is a three-euro cable.

Network disks (iSCSI)

Storage → Devices → Network disks.

A disk on the NAS that another machine sees as if it were its own: that machine formats it with its own file system (NTFS, VMFS, ext4) and one machine at a time uses it. It is what Windows Server, Proxmox and VMware ask for.

It is not a household folder: the NAS lends blocks and does not know what is inside. To share files between several people, use a household folder.

• The disk is a file inside the volume (<volume>/iscsi/<name>.img) and **takes up what is used**, not the full size from day one.
• With a password (CHAP) or without one. If you set one it has to be **between 12 and 16 characters**: that is what the Windows initiator accepts, and saying so here saves the "it will not connect and it does not say why" on the other end.
• Only these machines. You can say which machines may mount it, one per line. Leave it empty and any machine that reaches the port can. It is the second lock: a password can be passed around, a machine name does not travel.
• A disk that a machine has mounted cannot be removed. It would not give an error: it would corrupt whatever that machine was halfway through writing. For the same reason its list of machines cannot be changed while it is mounted either: disconnect it first from the machine using it.
• The panel keeps no list of its own: it shows the configuration the system will restore at boot, so it cannot say one thing while a server sees another.

Where a machine's name is

Every machine introduces itself with a long name starting with iqn. (its "initiator"). That is what you paste into Only these machines, and it is copied from the machine itself:

It looks like this: iqn.1991-05.com.microsoft:my-pc. Copy it whole, as is, without quotes.

How a Windows machine connects

1. iSCSI Initiator (it comes with Windows) → paste the address the panel shows (<nas-ip>:3260) → Quick Connect.
2. The disk shows up in Disk Management as a new unformatted one.
3. Initialise and format it like any other disk.

Measuring disk speed

Under Storage, “Measure speed” reads each disk and writes on each volume one at a time and saves the result next to each one (“180 MB/s on 8 Sep”). A disk measuring less than half of the previous time is marked in amber: the sign that it is going to fail before it fails. The test cuts off at 60 seconds per disk and is not run while a wipe or a disk replacement is running.