CCD Mirroring HOWTO

Introduction

This HOWTO describes how to set up partition mirroring on OpenBSD, using the concatenated disk driver, ccd(4). This is like RAID Level 1, but does not require special hardware or recompiling the kernel, and it works on individual partitions, not just on whole disks.

The official documentation for ccd is in the ccd(4) and ccdconfig(8) manpages. Before you ask any questions (especially on the mailing lists), make sure you have read and understood these, and also the ccd discussions on misc@, especially the threads CCD Mirroring HOWTO and Updated CCD Mirroring HOWTO.

Disclaimer

As always when you're messing with disks and filesystems, there is a risk of data loss. Make sure you have your valuable data backed up before you destroy it! I assume no responsibility for anything that happens to you, your computer, your data, your dog, your sanity, or anything else you can think of.

Quick Start

This section gives a short list of all the steps that need to be taken.

1. Create partitions on the disks you want to use. The partitions should have equal sizes, or you'll be wasting space.
2. Edit the file /etc/ccd.conf. As an example, to use wd0e and wd1e for mirroring, you can use the following line:

   
   ccd0	16	CCDF_MIRROR	/dev/wd0e /dev/wd1e
   

   Specifying other values than 16 might work, too, or even work better. I don't know; 16 works for me.
3. Run ccdconfig -C to enable the ccd0 device you just configured.
4. You can now partition the device with disklabel. You will first have to change the partition type of ccd0c from 4.2BSD to unused; after that, you can just partition the device as you would any disk.
5. Create filesystems on the newly created partitions. E.g. newfs /dev/rccd0a
6. That's all!

Why Use Mirroring?

It's a well-known fact that harddisks fail once in a while. When that happens, all data on them becomes inaccessible, except through expensive recovery methods. Making backups protects against data loss, but requires some effort, which is why many people don't do it. A backup is also usually behind the current state of the data, which means that all modifications between the time the backup was made and the time at which failure occurs is lost.

Mirroring provides a transparent way to keep multiple copies of the same data around. When one of the copies become inaccessible, the data can still be retrieved from another copy. Since the process is fully transparent, no effort is required to make backups, neglecting to make backups is impossible, and the mirrors are always current.

Of course, mirroring is no silver bullet. If you accidentally delete a file, it will be deleted on all mirrors. If all mirrors fail simultaneously, there are no mirrors left to recover the data from. If your last mirror dies before you replace the failed mirrors, your data is gone, too. For these reasons, it's important to make backups, even if you have your data mirrored.

How Does It Work?

ccd works by combining multiple partitions into a single virtual disk. There are three ways to combine these partitions: concatenation, striping, and mirroring. Concatenation simply creates a virtual disk that consists of all the component partitions after one another; it will have the combined size of all components added together. Striping means that the data on the virtual disk will be divided over the components. This way, a single blob of data is transfered to or from multiple disks at the same time, which can increase transfer speed. Finally, mirroring means that all components are copies of each other, so that if one fails, the data can still be retrieved from the others.

ccd is a pure software solution. The kernel takes care of the translation of operations on the ccd device to operations on the component devices. Because of this translation, the ccd device can be used just like any other disk. You can use disklabel to partition it just as if it were a regular harddisk. The only thing you can't do is use it outside of OpenBSD, because it doesn't exist outside of OpenBSD.

ccd vs. RAID

Those of you familiar with RAID have probably noticed the similarities between ccd and software RAID. In particular, the striping and mirroring provided by ccd are roughly the same as RAID levels 0 and 1, respectively. Having said that, there are also important differences. Most importantly, ccd doesn't provide the more complex RAID levels, and doesn't provide any automatic recovery. On the other hand, ccd is included in the GENERIC kernel, whereas a custom kernel needs to be built to use software RAID on OpenBSD.

Requirements

The only thing you need in order to be able to use ccd is a kernel that supports it. The OpenBSD GENERIC kernel (at least all versions up to 3.8) has ccd support built in, so you probably don't have to do anything special.

It would also be nice to have a couple of disks. This isn't actually necessary; ccd will happily work on anything that acts like a partition, so you could use it on a pair of mfs partitions, although it's difficult to see the utility of that. According to the manpage, you need an even number of components for mirroring to work. I don't see why it shouldn't work with 3 disks, but that's what it says.

You also need a computer, and some way to edit configuration files, and all sorts of other things that are are probably not necessary to list here.

Creating Components

A ccd device is created by combining a number of partitions. Thus, you need to create these partitions before you can create a ccd device. This is done with disklabel. The partition should have a type of ccd. On some systems, this type cannot be specified. In that case, specify the type as 4.2BSD.

On i386 systems, you may need to run fdisk to create an OpenBSD slice on the drive before you can run disklabel.

Creating the ccd Device

Once you have created the components, you can create the ccd device. This can be done with ccdconfig, but if you want the device to be present next time you boot the system, it's easier if you edit /etc/ccd.conf. The file takes lines of the following form:

device	ileave	flags	components

Here, device is the name of the ccd device you are creating, ileave is the interleaving factor, flags are the flags to be used with the device, and components are the component partitions of the ccd device.

The GENERIC kernel comes with four ccd devices available. Therefore, the possible values for device are ccd0 through ccd3. If you need more ccd devices, you will have to recompile the kernel.

The interleave factor has something to do with how data is spread over the components. If you set it to 0, you will get concatenation instead of mirroring. Other values indicate various forms of striping, and supposedly a good value for ileave leads to better performance when striping is used. All examples use the value 16, and since I don't know what the value does to mirroring, I've simply left it at that.

flags must be set to CCDF_MIRROR to enable mirroring. For striping or concatenation, it's typically set to none.

Components is the list of partitions that make up the ccd device.

As an example, to combine the partitions /dev/wd0e and /dev/wd1e into ccd0, with mirroring, use:

ccd0	16	CCDF_MIRROR	/dev/wd0e /dev/wd1e

Note that the ccd device is not preceded by /dev, and the component partitions are.

When you're done editing the file, save it and run ccdconfig -C (that's an uppercase C). This will create all ccd devices listed in the configuration file. If something goes wrong, you will see an error here. If all is well, the command outputs nothing. By the way, this very same command is run at boot time to create the ccd devices. This implies that the root filesystem must already have been mounted. Hence, you cannot put your root filesystem on a ccd device.

Labeling the ccd Device

Once the ccd device has been created, you can use disklabel to create partitions on it. A c partition of type 4.2BSD spanning the whole device will have been created. You will need to change the type of this partition to unused before you can create any partitions, otherwise disklabel will complain about there being no space left.

The initial version of this HOWTO stated that you could also just use the c partition that was created for you. Although this seems to work for me and others, some people have warned that dire consequences will ensue if you go this way. To be safe, just set the c partition to unused and create partitions, just like you would on any real harddisk.

Creating Filesystems

Now that the ccd device has been labeled, you can create filesystems on it. This follows the normal procedures for creating filesystems. Suppose you have created a device ccd0 and created an a partition on it. You could then create a filesystem on it by running:

newfs /dev/rccd0a

If you used a different ccd device or want to create a different type of filesystem, substitute appropriately.

Mounting

With the filesystems created, you can mount the appropriate partitions like any other partition. For example, if you had followed the example from the previous section, you could mount the filesystem by running:

mkdir /mnt/ccd0a
mount /dev/ccd0a /mnt/ccd0a

Of course, you can also add a line to /etc/fstab to get your filesystem automatically mounted at boot time.

FAQ

Can I set up ccd on a partition without losing the data on it?

The short answer is no. To create a ccd device, you need to dedicate a partition to it and create a disklabel on that partition. This last step alters what's on the partition and will cause your data to be lost.

One way to get a ccd device that has the same data on it as your partition is to first make a dump of that data, then restore it after you've created the ccd device.

For example, assuming that /home on wd0k is the partition you want to turn into a ccd device, and /mnt/scratch contains a filesystem with enough space on it to hold the dump, you could take these steps:

1. Mount /home read-only, so that you get a consistent dump:

   # mount -u -o ro /home

2. Dump /home:

   # dump -a -f /mnt/scratch/home.dump /home

3. Create the ccd device, initialize it, and mount it on /home.
4. Change directory to /home:

   # cd /home

5. Restore the dump:

   # restore -r -f /mnt/scratch/home.dump

Can I use ccd on my root partition?

Not really. ccd devices need to be configured by running ccdconfig before they can be used. Normally, ccdconfig -C is run from /etc/rc. This means that all of /sbin/ccdconfig, /etc/ccd.conf and /etc/rc must be present, and these are on the root filesystem.

You could get around this limitation by making a very small root partition with only enough on it to configure your ccd devices, mount your real root partition from there, and then chroot to the real root partition and continue booting. However, general consensus is that this is too much trouble.

If you want to have your root filesystem duplicated, the suggested mechanism for this is altroot. The afterboot(8) manpage contains some information on setting this up.

Why is my question not answered in this FAQ?

Probably because it hasn't been asked often enough. In any case, contact me with any questions you may still have.