Blog: How-Tos

How to make a software BTRFS RAID1 with LUKS2 FDE

Pedro Venda 22 Dec 2020

The guide below is simplified in a way that preparing the boot partition is not covered.

Software based btrfs RAID1 requires two devices, which conceptually don’t even need to be on different disks. But for obvious reasons, it’s a good idea if they are…

Having mirroring against encrypted storage brings its own set of challenges: It’s easy to encrypt a mirrored device (e.g. using md whereby each device has exactly the same data blocks), then use LVM and file systems on top. It’s not so easy to have this done if the mirroring is done at a layer “higher” than the encryption, since it is necessary to deal with multiple individually encrypted devices. Which is fine. More complicated but certainly not impossible.

This post aims to guide through implementing btrfs mirroring on two encrypted storage devices.

Disk partitioning

The basic requirement for btrfs RAID1 is having two partitions, ideally in different disks, as described below:

# fdisk -l /dev/sda
Disk /dev/sda: 978.1 GiB, 1050214588416 bytes, 2051200368 sectors
Disk model: Crucial_CT1050MX
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: D2E25ADE-25A7-F056-9982-F40C48B55D53

Device Start End Sectors Size Type
/dev/sda1 2048 526335 524288 256M EFI System
/dev/sda2 526336 1574911 1048576 512M BIOS boot
/dev/sda3 1574912 2623487 1048576 512M Linux filesystem
/dev/sda4 2623488 935856127 933232640 445G Linux filesystem

Similarly, there was a /dev/sdb4 on the second disk which had the same size. For btrfs, it is not a requirement that both devices have the same size, since the RAID setup will take as much space as the smallest device allows. Since LVM2 will be in place, all components of the system will live under /dev/sda4 and /dev/sdb4 as logical volumes sitting on encrypted physical volumes.

The disks in this example were partitioned with a GPT label. This is helpful but not required for the purpose of this howto.

Lastly the boot partitions (/dev/sda3 and /dev/sdb3) were equally configured in a btrfs raid1 layout, except these were not under LUKS2 encryption.

LUKS2 encryption and boot tricks

Before considering the file system, the partitions need encrypting using cryptsetup to implement luks2.

For the purpose of this tutorial, and for the boot process to be simple, both devices are encrypted with the same passphrase. This means the actual keys are unique per device but the passphrase used to decrypt the storage encryption keys is the same on both devices. This is optional, just makes life easier.

Volume names ‘crypt-d1’ and ‘crypt-d2’ were chosen, which will tie in with /etc/crypttab’s content.

# cryptsetup luksFormat /dev/sda4 --type luks2
# cryptsetup luksOpen /dev/sda4 crypt-d1

# cryptsetup luksFormat /dev/sdb4 --type luks2
# cryptsetup luksOpen /dev/sdb4 crypt-d2

At this point, it’s worth noting down the respective partition UUIDs and creating /etc/crypttab

# echo "crypt-d1 PARTUUID=$(lsblk /dev/sda4 -o partuuid -n) btrfs_r1 luks,keyscript=/lib/cryptsetup/scripts/decrypt_keyctl > /etc/crypttab

# echo "crypt-d2 PARTUUID=$(lsblk /dev/sdb4 -o partuuid -n) btrfs_r1 luks,keyscript=/lib/cryptsetup/scripts/decrypt_keyctl >> /etc/crypttab

Beyond the automation, here’s what /etc/crypttab looks like:

# cat /etc/crypttab
crypt-d1	PARTUUID=ab4f59d1-8e44-2839-8e8f-3b42e2db5192	btrfs_r1	luks,keyscript=/lib/cryptsetup/scripts/decrypt_keyctl
crypt-d2	PARTUUID=aa3a4e05-84b7-b758-b66f-bbafa1271f6a	btrfs_r1	luks,keyscript=/lib/cryptsetup/scripts/decrypt_keyctl

The above lines under /etc/crypttab have the following impact:

labels crypt-d1 and crypt-d2 as ‘btrfs_r1’ for the purpose of handling by cryptsetup tools;
Once the passphrase has been entered for crypt-d1, it’s handled by the decrypt_keyctl to assign transparently to crypt-d2 (and any other device matching the ‘btrfs_r1’ label);

LVM2

Now that the encrypted volumes exist and are available, LVM is used to manage the encrypted space. Each encrypted device will become a physical volume (PV) and hold its own volume group (VG) and logical volumes (LV).

# pvcreate /dev/mapper/crypt-d1
# vgcreate disk1 /dev/mapper/crypt-d1
# lvcreate -l 90%VG -n root disk1
# lvcreate -l 100%VG -n swap disk1

### repeat for PV crypt-d2 into a VG 'disk2' ###

After setting up, the layout should be similar to the following:

...
├─sda4 8:4 0 445G 0 part
│ └─crypt-d1 253:0 0 445G 0 crypt
│   ├─disk1-swap 253:1 0 16G 0 lvm [SWAP]
│   └─disk1-root 253:2 0 429G 0 lvm /
...
└─sdb4 8:20 0 445G 0 part
| └─crypt-d2 253:3 0 445G 0 crypt
|   ├─disk2-swap 253:4 0 16G 0 lvm [SWAP]
|   └─disk2-root 253:5 0 429G 0 lvm
...

You will notice that until this point both disks have been setup individually, each with their own encrypted space and LVM layout.

# pvs
  PV                   VG    Fmt  Attr PSize   PFree
  /dev/mapper/crypt-d1 disk1 lvm2 a--  444.98g    0 
  /dev/mapper/crypt-d2 disk2 lvm2 a--  444.98g    0

# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root disk1 -wi-ao---- 428.98g
swap disk1 -wi-ao---- 16.00g
root disk2 -wi-ao---- 428.98g
swap disk2 -wi-ao---- 16.00g

btrfs RAID1 filesystem

It is at this point that btrfs makes an appearance. First by being created under /dev/mapper/disk1–root, then by being extended under RAID1 into /dev/mapper/disk2-root.

# mkfs.btrfs /dev/disk1/root
# mount /dev/disk1/root /mnt/root
# btrfs device add /dev/disk2/root /mnt/root
# btrfs filesystem label /mnt/root root-r1
# btrfs filesystem balance start -dconvert=raid1 -mconvert=raid2 /mnt/root

The above sequence of commands creates a btrfs filesystem under /dev/disk1/root, labels it as ‘root-r1’, adds the device /dev/disk2/root and balances the file system across the two devices in a raid1 configuration.

Checking the btrfs RAID filesystem can be done using btrfs filesystem usage and btrfs device usage as shown below:

# btrfs filesystem usage /
Overall:
    Device size:         857.96GiB
    Device allocated:        258.06GiB
    Device unallocated:      599.90GiB
    Device missing:          0.00B
    Used:            248.08GiB
    Free (estimated):        303.13GiB  (min: 303.13GiB)
    Data ratio:               2.00
    Metadata ratio:           2.00
    Global reserve:      138.28MiB  (used: 0.00B)

Data,RAID1: Size:127.00GiB, Used:123.82GiB
   /dev/mapper/disk1-root    127.00GiB
   /dev/mapper/disk2-root    127.00GiB

Metadata,RAID1: Size:2.00GiB, Used:226.30MiB
   /dev/mapper/disk1-root      2.00GiB
   /dev/mapper/disk2-root      2.00GiB

System,RAID1: Size:32.00MiB, Used:48.00KiB
   /dev/mapper/disk1-root     32.00MiB
   /dev/mapper/disk2-root     32.00MiB

Unallocated:
   /dev/mapper/disk1-root    299.95GiB
   /dev/mapper/disk2-root    299.95GiB

# btrfs device usage /
/dev/mapper/disk1-root, ID: 1
   Device size:           428.98GiB
   Device slack:              0.00B
   Data,RAID1:            127.00GiB
   Metadata,RAID1:          2.00GiB
   System,RAID1:           32.00MiB
   Unallocated:           299.95GiB

/dev/mapper/disk2-root, ID: 2
   Device size:           428.98GiB
   Device slack:              0.00B
   Data,RAID1:            127.00GiB
   Metadata,RAID1:          2.00GiB
   System,RAID1:           32.00MiB
   Unallocated:           299.95GiB

Local fstab

For the purpose of fstab, both /dev/disk1/root and /dev/disk2/root will have the same file system UUID so either can be mounted. However, using the UUID will solve the problem transparently.

Conceptually, after swap space has been created under /dev/disk1/swap and /dev/disk2/swap (not RAID), then root and swap entries on fstab can be created as follows:

echo "
UUID=$(lsblk /dev/disk1/root -o uuid -n) / btrfs defaults 0 0
UUID=$(lsblk /dev/disk1/swap -o uuid -n) none swap sw 0 0
" >> /etc/fstab

Notice that under fstab UUID is used, whereas under /etc/crypttab PARTUUID was used instead. Only one UUID was necessary to reference the root file system (not both). Also notice that since PARTUUID and UUIDs are used, there’s no fiddling with device names or locations – if you’re not using UUIDs instead of device paths, do it, it will change your life. Here’s what fstab looks like:

# cat /etc/fstab
UUID=	/boot	btrfs	defaults	00
UUID=	/boot/efi	vfat	defaults,noatime	0	0
UUID=e76a6d38-096c-301b-f56b-ab0d7170f725	/	btrfs	defaults	00
UUID=ad711836-473e-2449-2bde-5419c7cef40f	none	swap	sw	0	0
UUID=316d8414-371f-6910-3b5f-cce6b1dd0bf9	none	swap	sw	0	0

Boot loader and initramfs

Ensuring that the boot loader and initramfs are setup correctly are not entirely trivial matters. If the above has been done on a running system

with existing /boot partition (with bootloader files)
with correctly labelled LUKS2 volumes (crypt-d1 and crypt-d2)
with correct LVM names (disk1 and disk2)
with correctly constructed /etc/fstab and /etc/crypttab,

then running the following sequence of commands should enable preparation of the initramfs with the correct cryptsetup binaries and helper scripts (decrypt_keyctl) as well as correctly constructed grub.conf.

# update-initramfs -u
# update-grub
# grub-install /dev/sda
# grub-install /dev/sdb

How to make a software BTRFS RAID1 with LUKS2 FDE

Disk partitioning

LUKS2 encryption and boot tricks

LVM2

btrfs RAID1 filesystem

Local fstab

Boot loader and initramfs

Suffered a Security breach?

Mobile Security

Social Engineering

Web application testing

Security Consuting

Papa - PTP Advance Password Auditor