SSD, LVM and you: Zero data loss, LVM caching, and properly configuring your Linux box to get the most out of SSD!

I recently upgraded tesla, my primary Linux desktop, to one of the new Ryzen processors from AMD.  So far it’s working very well, and the performance is sometimes even rather stellar.  One of the big performance bosters, in addition to the Ryzen CPU and doubling the memory, was I went SSD in a big way.

Now, it’s important to note, that I have a rather large music collection and some videos, a crap load of pictures, and other media I don’t want to store on SSD as it would be prohibitively expensive at this point in time.  So, the system, before upgrading had only a old smallish SSD that only had /boot and / on it.  And a 4 TB platter that had /var, /home, and swap on it.

The new configuration has a M.2 drive with /boot, /boot/efi, and / ( including /var this time ) on it. There is also a SATA 3 SSD with /home on it. swap, and what I’ll call /home2 is the platter.

An aside note about swap. Yes, tesla now has 32 gigs of ram, but the way PC processors, and Linux memory management works, it’s better to have some swap than none at all, even if you don’t expect it to be used. General rule of thumb, is 2x Memory to 8 Gig, then same as memory until you go over 16 gig, the 16 gig. I had 16 gig memory, and the new on will have 32, so I already have a perfectly good 16 gig swap partition. that is being used.

And a little disclaimer: This stuff worked very well for me. It may not for you. Make backups before trying ANY of this! I have backups. This was not a concern for me. If you don’t have backups and loose data, it’s my fault. You’ve been warned.

Now, for the pre-work. The VG is the default fedora. I like to keep everything in on VG. It makes life easier.  First off, the home lv needs to be renamed to home2.  So boot the system into single-user mode, and rename the lv:

lvrename /dev/fedora/home /dev/fedora/home2

Next, the VG has to be exported, since it’s fedora, the system has to be booted into recovery mode, then export it:

vgexport fedora


This is what I came up with for a  filesystem layout:

Filesystem Drive Size
/boot M.2 1 GB
/boot/efi M.2 1 GB
/ M.2 100GB
SWAP Platter 16 GB
/home2 Platter Rest
Cache M.2 40 GB
Cache Meta M.2 256 M


As I was putting it all back together, I left the 4TB disk disconnected and put the M.2 drive and left the other SSD unplugged to prevent confusion during the install. I did a normal XFCE spin install, except I did a custom disk config only using an LVM scheme and only putting /boot, /boot/efi, and / on the M.2 drive as described in the table above.  A note here, if I plugged the 4 TB disk in at this point, the system would not boot.  You’ll have to plug it in hot. when the time comes. Not ideal, but it worked for me.

Once that was done, I connected the SATA SSD for /home, and added it:

​fdisk /dev/sdb
pvcreate /dev/sdb1
vgextend fedora /dev/sdb1
lvcreate -n home -l 100%PVS fedora /dev/sdb1
mkfs.ext4 -j /dev/mapper/fedora-home

That was the easy part, now to import the 4 TB disk, as I really don’t want to have to restore from disk.  Plug it in, and read some man pages.  Turns out, it’s actually easy.

First import it the existing PV:

vgimportclone --import /dev/sda1

This gives a VG called fedora1.  Close, but not quite what I want.

So offline it, then merge the two together:

vgchange -a n fedora1
vgmerge -v fedora fedora1

Now I have on nice big VG, fedora, that has all three drives in it. Perfect!

Now a little cleaning up to get rid of the old /var, expand home, and it’s probably a good idea to refresh the swap space:

lvremove /dev/fedora/var 
lvextend -l +100%PVS /dev/fedora/home2 /dev/sda1
resize2fs /dev/fedora/home2 
mkswap /dev/fedora/swap

And finally for this part, edit fstab, make the mount points, and mount everything:

mv /home /home.orig
mkdir /home
mkdir /home2
vi /etc/fstab
mount -a
swapon -a

NOTE: When configuring the fstab, make sure to add noatime to the mount options list for all the SSD’s.  It helps with their life and performance.

Next up: configure fstrim. This is as easy as making a script that runs /usr/sbin/fstrim -a -v and putting it in /etc/cron.weekly.  There are other ways to do this, but this seems to be the preferred.  Then set issue_discards = 1 in /etc/lvm/lvm.conf. That should be picked up on the next reboot.

Now set up the elevators.  This is done with a script that sets the deadline queue in /sys/block/{ssd}/queue/scheduler.  Then a systemd unit is created to run it at boot time.

You can find the scripts and systemd units as well as my script to run fstrim and other possibly interesting things on my ‘system’ github repo:

Remember to enable and start them everything is installed.

And the final piece of the puzzle, using some spare space on the M.2 drive for LVM caching for the platter.  This isn’t that hard, but took some reading to figure it out.

The installer only allocated the space I requested to the PV for root. So the easiest way to fix that is just create another PV on the M.2 device using the rest of the space:

fdisk /dev/nvme0n1
pvcreate /dev/nvme0n1p4
vgextend fedora /dev/nvme0n1p4

Now to create the cache pool and meta device.  Since I’m working on a 4 TB disk, it seemed to make sense to make the cache pool 40 G.  I made the Meta 256 Meg, probably over kill ( like my /boot/efi ), but I don’t want to have to worry about resizing it in the future:

lvcreate -L 256M -n home2_cache_meta fedora /dev/nvme0n1p4 
lvcreate -L 40G -n home2_cache fedora /dev/nvme0n1p4 
lvconvert --type cache-pool --poolmetadata fedora/home2_cache_meta fedora/home2_cache

Now just attach it the lv:

lvconvert --type cache --cachepool fedora/home2_cache fedora/home2

And after that, just give it a reboot and make sure everything comes up as expected.

You can use vgdisplay, vgs, and pvs to verify everything is the way you want it.

Comments 2

  • Heya, I also have a Ryzen (7 2700) with 32 GB of ram. Also thinking of grabbing a nvme, but have 2 ssds for now. Can you give some feedback on how this has worked out for you.. the cache, i/o patterns over the hierarchy (spindlecachessd<-? nvme ) etc.?


    Giblon 5 out.

  • It’s been fine. Performance is good. Writes to the platter are super fast. Reads can be quick depending on the data ( streaming files for example get pre-cached when opened – so as long as there is enough cache space for them, they are quick ).

    Only thing I notices is under heavy reads, there is a lot of disk activity on /home2, like during backups and when locate db builds. The other oddity is that, unless it’s buried in IO and on constantly, the hard drive light on my PC makes no sense. It just kind of flashes what seems to be randomly. Even when no disk is likely to be accessed.

Leave a Reply