LVM2

LVM: Logical Volume Management, Version 2.

NOTE: Version 2 is recommended if you are running a 2.6 kernel.

tldp.org/HOWTO/LVM-HOWTO

What is it?

Normally /dev/hda is a drive, and /dev/hda1 is a partition of that drive. In LVM, those are called Physical Volumes (PV).

You put this PVs into a Volume Group (VG). From that Volume Group, you carve out Logical Volumes (LV). LVs act like partitions.

so it looks like this:

    hda1   hdc1      (PVs on partitions or whole disks)
       \   /
        \ /
       diskvg        (VG)
       /  |  \
      /   |   \
  usrlv rootlv varlv (LVs)
    |     |       |
 ext2  reiserfs   xfs (filesystems)

Requirements

2.4 kernel with device mapper patch, or 2.6 kernel with device mapper enabled. The stock debian 2.6 kernels have device mapper support.

Install the lvm2 user tools:

 # apt-get install lvm2

Setting it up

Create Physical Volume

For a partition:
bc. # pvcreate /dev/hdb1

Or for a whole disk:

 # pvcreate /dev/hdb

If adding an entire disk to a physical volume, the disk must not have a partition table.

Create Volume Group

 # vgcreate testvg /dev/hda1 /dev/hdb1

Test to make sure it worked:

 # vgdisplay testvg

Create Logical Volumes

To create a 1500MB linear LV named ‘testlv’:

 root> lvcreate -L1500 -ntestlv testvg

Test to make sure it worked:

 root> lvscan -v

Create the filesystem:

 root> mke2fs /dev/testvg/testlv

Mount the new LV:

 root> mount /dev/testvg/testlv /mnt

Using it

Reducing size

to reduce spool by 1 gig:

  umount /dev/sympavg01/spool
  resize_reiserfs -s -1G /dev/sympavg01/spool
  lvreduce -L -1G /dev/sympavg01/spool
  mount -treiserfs /dev/sympavg01/spool /home/sympa/spool

what this is doing:

  1. unmount the device!
  2. make filesystem by 1 gig smaller
  3. make volume 1 gig smaller
  4. mount again

Increasing size

  lvextend -L +1G /dev/sympavg01/archives
  resize_reiserfs -f /dev/sympavg01/archives

what this is doing:

  1. make the logical volume 1 gig bigger
  2. grow filesystem to fill the volume size

With an ext2/3 filesystem, it might be like this:

  lvextend -L +300M /dev/mirror_vg/var_lv
  resize2fs /dev/mirror_vg/var_lv

This will grow /dev/mirror_vg/var_lv by 300 megabytes, and then make the ext2 or ext3 filesytem on that device grow to fill the device size.

removing a volume

lets say you have /dev/hda and /dev/hdb that are a part of a volume and you want to take
out /dev/hda if you have enough space on /dev/hdb you can issue a command which will move
all data off of /dev/hda (pvmove /dev/hda) that will move everything from /dev/hda onto
the rest of the disks in the volumegroup (in this example /dev/hdb)… it isn’t a fast
thing as it is a block copy, but you can see what is going on with a -v… then once it
is removed you can remove the disk from the volume group with a vgreduce then you can put a new disk in and redistribute the moved data to the new disk

copying over an existing partition

ok, this is not exactly an lvm issue, but it comes up a lot.

Lets suppose we have a new LV which we want to become our “usr” partition.

  1. mount the new partition on /mnt
  2. kill off any process which has files locked on /usr
  3. use find and cpio to transfer files from /usr to /mnt
  4. remove files in /usr/
  5. remount /mnt as /usr

it might look like this:

# mount /dev/vg1/usrlv /mnt
# lsof /usr
# kill <some pids>
# cd /usr
# find -xdev | cpio -vdamp /mnt
# umount /mnt
# cd /usr
# rm -r *
# cd /
# mount /dev/vg1/usrlv /usr

FAQ

This is a list of questions that I’ve seen come across the LVM mailing list
that deal with everything from how do I restore this LVM partition that
failed because one of the underlying disks died, to how to recover from
failures etc. It is mostly a random assortment of things that I’ve pulled
together, and is still a work in progress.

How to use a LVM devices on a different machine:

If you have LVM setup on disks on machine A, and it goes down and you want to take the drives out and put them in machine B, what do you need to do to prep machine B beforehand to deal with this? You don’t need to do anything apart from vgscan/vgchange -ay when

you need to use the volumes, then mount them. vgscan/vgchange reads
the metadata off the disk and creates the necessary volume groups and
device files on the system. In otherwords, /dev/lvg1/lvol1 will be
created on machine B based on the metadata. Do not re-do vgcreate or
lvcreate because this will mangle the meta data for the other node.

A disk has died, how do I get the data off?

Try dd_rescue www.garloff.de/kurt/linux/ddrescue (either to another disk of the same size, or to a file which can be burned to a CD or mounted in a loopback)

How do I build a mirrored LVM system with Debian?

  1. Start Debian install
  2. Create 20MB /boot filesystem/partition at the start of the disk.
  3. Then create a “temporary” root filesystem/partition the size that you want swap to eventually be (minimum 200MB or so to get minimal debian install onto it)
  4. Then create one partition for all remaining space
  5. Do not specify to mount /boot partition during installation so that /boot files are put on “temporary” root filesystem
  6. Do minimal Debian install (no to tasksel and dselect) into 250MB partition, specify no swap partition
  7. When all done installing truly minimal install, put second disk
  8. Use sfdisk to duplicate partition table to second disk
  9. Create raidtab mapping partition(s) 1, 2 and 3 to md0, md1 and md2 respectively
  10. mkraid /dev/md0
  11. Mount /dev/md0 on /mnt and copy /boot to it, remove /boot/* and mount /dev/md0 on /boot
  12. mkraid /dev/md2
  13. pvcreate /dev/md2
  14. vgcreate /dev/md2
  15. lvcreate root, mkfs … /dev/…/root
  16. lvcreate usr, mkfs … /dev/…/root
  17. lvcreate …
  18. mount /dev/…/root /mnt, /dev/…/usr /usr, …
  19. copy -rax / /mnt, cp -rax /usr /mnt, …
  20. Create initrd to reflect above — I use Adrian Bunk’s initrd-tools
  21. Edit /etc/lilo.conf to reflect /dev/md0 is boot partition and lvm lv as root partition
  22. edit fstab to reflect above changes
  23. reboot
  24. mkraid /dev/md1
  25. add /dev/md1 to fstab as swap partition and swapon -a

My system can’t find the volume groups!

vgscan tells me no volume groups are found
try lvscan
lvscan — ERROR: “/etc/lvmtab” doesn’t exist; please run vgscan
vgscan — ERROR “creating lvmtab directory” creating lvmtab
vgscan — no volume groups found
mount -o remount,rw /
then check for /etc/lvmtab and /etc/lvmtab.d
ls: lvmtab.d: Input/output error
boot the machine with a boot disk and fsck root! Or do touch /forcefsk
and reboot single-user. Fsck says its clean? Try fsck -f
do pvscan
inactive pv hda10 is associated to unknown volumegroup
sympavg01 (run vgscan)
run vgscan, this will create lvmtab.d
then run lvscan
volumegroup is not active try lvscan -D, do a vgchange -a y
(instead of lvscan -D because lvscan -D will only scan, not activate)

When setting up a volume group, half of the space is gone, whats up
with that?

When you do the vgcreate command it says how large the physical

extent sizes are, and there is always one extent that is reserved at
the beginning of the partition for the LVM metadata. So if your
physical extent size is 32MB, then you will lose 32MB for the
metadata. Also, LVM volumes have to be multiples of the extent size
(32MB, 64MB, 96MB, etc.), so if you make a volume that is 93megabytes,
that is 3MB too little for a 3 extent volume, which leaves you with a
2 extent volume totalling 2 * 32 MB = 64MB.

What is the maximum size of logical volumes?

It depends on which version of LVM you are using. If you are
using LVM1, then there is a 255G limit on LV sizes (this is based on
PE size). If you use the
LVM2 metadata format, then you are limited only by disk space and the
maximum size that your kernel supports. If you create PVs in LVM1
format and then migrate to LVM2, then you will still be limited to
255G, unless you remake your VGs and LVs with the LVM2 tools. If you
made a VG with LVM1, or if you created it with the -M1 flag to
vgcreate in LVM2, then you have an upper limit which is imposed by the
format1 metadata.

What filesystem is best with LVM?

It all depends on what you need. All the different filesystems
work under LVM, it would be best to do some research into the current
technologies that each offer, and make an assessment based on what
your needs are. There are a number of benchmarks out there, and the
filesystems are changing rapidly, so these benchmarks will get out of
date and need to be updated.

Here are some benchmarks:
otn.oracle.com/oramag/webcolumns/2002/t...
oregonstate.edu/~kveton/fs
fsbench.netnation.com

Ext2: The most stable filesystem out there for Linux is ext2, think twice
about not using it, it actually is fast and has what most people need.
Seriously consider the reasons why you want to use something else,
because most journaling filesystems are still very much in a
development stage and are risky. Stick with ext2 if all you are
wanting is the ability to never fsck again, because this is not
something you will gain with a journaling filesystem. Disorderly
shutdowns, even with full journaling on your filesystems, can lead to
dataloss of all data that is stored in the buffercache, however
filesystem consistancy will probably be maintained.

Ext3: This is one of the most stable journaling filesystems out as of
this writing. It has been in the kernel for some time, and is mostly
just a hack ontop of ext2. It is simple for converting your ext2
systems to ext3 and back, but it requires some kernel patches if you
want to do online resizing. If you are using Redhat, probably should
stick to ext2/3, Redhat does that well and doesn’t do others so well,
so you are fighting if you do anything else (in fact the ext3 online
extend code is included in the latest RedHat Rawhide kernel).

Reiserfs: They say reiserfs has much better performance in dealing
with lots of small files, but it doesn’t have any bad block management
capabilities. If your drive gets bad blocks, you can very quickly be
rebuiling your entire filesystem from backup. Reiser currently performs rather
poorly on really large files (ie. databases). Oracle doesn’t support
the tuning of a system if it is running reiserfs, and ranks the
different filesystems as follows: ext2, ibm-jfs, ext3, reiser. Reiser
works great for maildir spools and usenet feeds. Because resier3 does
not do true data journaling (only metadata), it is able to bring
filesystem integrity back to normal after a bad shutdown, but some
data loss is possible. Reiser4 supposedly will fix this (as well as
add plug-ins). It seems a lot of people avoid Reiser on Redhat
(something having to do with redhat’s kernel interoperability causing
problems, they support ext3 well, so I would recommend sticking with
ext3 if you are using Redhat), but a lot of SuSE people use it, as it
is well integrated. I personally am a fan of Reiser for maildir spools
and list archives that are parceled out in very large individual
files, it has been pretty stable for us so far, although the website
graphics freak me out and the advertising you get when you create a
filesystem is always a bit of a shock (and “secret” sources make me
nervous too).
On that note, if you are using reiserfs, it balks badly at being a
readonly filesystem. Even if you mount it readonly, it still wants write
access to the journal. We have a situation in our shop where linux can not
access the disks in read/write mode. For those filesystems that were
bootable, we cannot use reiserfs.

JFS: You can dynamically grow this filesystem without intervention,
which is kinda neat, but scary. I have yet to see people consider JFS
stable in linux

XFS: ? In terms of stability, I’ve had significant problems with it in
the past, but it could’ve been a particular kernel patch/version. XFS
only made it into the official kernel in 2.4.23, so be aware…

pvmove gives “device-mapper: error adding target to table” errors

Do you have the device-mapper mirror target installed? Not
having it available is the most common cause of this problem. Some 2.6
kernels don’t include this.

  • I cannot mount my snapshot volume, gives “wrong fs type, bad option,
    bad superblock on /dev/etc or too many mounted filesystems” *
    Try mounting with the parameters “-onouuid,ro”

My logical volumes are inactive and unusable!

Run “vgscan” and then “vgchange -ay” to activate all the VGs and the LVs.

My logical volumes are not active or usable on restart!

Check that you have your startup scripts available that start your
logical volumes. You should have one that does a “vgscan” and then a
“vgchange -ay” somewhere at bootup, without this your VGs and LVs wont
be available on bootup.

The system is not recognizing my boot LVM partitions and kernel panics.

You need to run “lvmcreate_initrd”, which will create an
lvm-initrd file in /boot. Then edit grub.conf to use the new initrd.
This will make your lvm tools available at boot and your VGs will come
online. To do this with a rescue disk do the following (change
/mnt/sysimage to whatever your actual root is mounted as):

		mount -t proc proc /mnt/sysimage/proc
		chroot /mnt/sysimage /bin/bash
		source /etc/profile
		lvmcreate_initrd
		exit
		umount /mnt/sysimage/proc
		sync
		shutdown -r now

How do I see if I have LVM support available?

LVM1: cat /proc/devices and see if you have “lvm” listed, see if
you have the vgcreate command.

LVM2: check to see if you have the “lvm” base command by doing
“lvm version” – this should give you complete output and confirm that
device-mapper is available in the live kernel and that
/dev/mapper/control is setup correctly (see
device-mapper/scripts/devmap_mknodsh). You can look in /proc/misc to
see if there is device-mapper support in the running kernel.

How do I go from LVM1 to LVM2?

If you have a LVM1 setup on your Debian machine you just need to
switch to LVM2 by installing the lvm2 packages. lvm-common is the
Debian package that switches between LVM versions on the same system
You will get this package as a dependency when you install lvm10 or
lvm2. The Debian LVM binaries get installed libraries in
/lib/lvm- and lvm-common decides at run-time which one is
most appropriate to use.

How do I go from a 2.4 kernel to a 2.6 kernel?

You need to make sure you have CONFIG_DM_IOCTL_V4 support in your
kernel for device-mapper. You need to select CONFIG_BLK_DEV_DM so that
you can see the option for IOCTL_V4

How do you get device mapper in the 2.4 kernels?

Follow the directections for device mapper at evms.sourceforge.net/install/kernel.html

Glossary

LVM1 = Userspace tools + kernel ioctls included in marcelo’s 2.4 tree

  • LVM1 kernel ioctls are not included in or available for 2.6
  • LVM1 userspace tools do not work with 2.6 kernels

dm = Kernel driver (GPL) for new volume managers to use.

  • Included in Linus’s 2.6 kernels.
  • Available as a patch for 2.4 kernels from the Sistina website.
  • Knows nothing about volume manager’s on-disk metadata layouts.
  • Userspace volume managers (e.g. EVMS and LVM2) communicate via a new
    ioctl interface.
  • This ioctl interface is currently “version 4” and we regard it as
    stable. binaries.
  • An old development version of this device-mapper ioctl interface known
    as “version 1” has problems with it, is deprecated and should be
    removed from kernel trees ASAP.
    Always use “version 4” when building new kernels today.

libdevmapper = Userspace shared library (LGPL) which wraps a volume manager application interface around the device-mapper ioctls

  • Can determine transparently whether the kernel device-mapper is using
    “version 4” dm ioctl interface or the deprecated “version 1” interface
    and adapt itself accordingly. configure --enable-compat
  • Can only communicate with device-mapper: it cannot use LVM1 ioctls.
  • Designed primarily for use by LVM2 tools. EVMS does not use it
  • Some parts of the libdevmapper API are not yet stable and are likely
    to get changed.

dmsetup = Userspace utility (GPL) which provides full command-line access to the libdevmapper API.

  • Designed for use by shell scripts and for testing and debugging.
  • Command line interface may be considered stable. New features may get
    added, but we’ll try not to break existing commands.

LVM2 = New Logical Volume Manager command line tools (GPL) designed to be backward-compatible with LVM1 and offering new features and more flexibility, configurability and stability.

  • Supports existing LVM1 on-disk metadata.
    This means you do not have to make changes to your existing on-disk
    LVM1 volumes to switch between using LVM1 and LVM2.
  • Uses command lines similar to LVM1.
  • By default uses a new on-disk metadata format supporting more
    features than the original LVM1 version.
  • Communicates with the device-mapper kernel driver via libdevmapper’s
    API.

Miscellaneous points:

  • LVM1 uses block major number 58: dm selects one or more major numbers
    dynamically as required instead.
  • LVM1 uses character major number 109: dm selects a misc minor number
    dynamically instead.
  • There’s a (non-devfs) script for creating /dev/mapper/control at
    startup (or after dm module load).
  • You can use LVM1 tools with unpatched 2.4 kernels.
  • You can use LVM2 tools with patched 2.4 and unpatched 2.6 kernels.
  • Device-mapper support for snapshots and pvmove is so far released
    only for 2.4. Patches for 2.6 are being tested.
  • Multipath and mirror support are under development for 2.6.
    (Then get back-ported to 2.4.)

Web download page: www.sistina.com/products_lvm_download.htm

The device-mapper tarball contains:
device-mapper kernel patches – needed only for 2.4;
userspace libdevmapper and dmsetup – needed with all dm kernels.
The LVM2 tarball contains the LVM2 command line tools.

Development code can be found via:
people.sistina.com/~thornber (for kernel patches)
www.sistina.com/products_CVS.htm (for userspace code)

Device-mapper mailing list:
lists.sistina.com/mailman/listinfo/dm-d...