Solaris ATA-over-Ethernet (AoE) Installation and Operation Guide

This manual describes Version 1.4 of the Solaris ATA-over-Ethernet subsystem, which makes EtherDrive®¹ Storage Blades available as disk devices. It explains how to install and configure the AoE package; how the disks appear to the rest of the Solaris system; and some quirks and implementation details. It is not a manual on how to install EtherDrive hardware.

Parts of the AoE subsystem work differently if the Solaris System Management Facility (SMF) is in use (svcs, svcadm, and svccfg(1M)). Solaris 10 normally uses SMF; Solaris 9 and earlier systems don't have it. When the difference matters, this manual will refer to SMF and non-SMF systems.

Parts may also behave differently on the slightly-different versions of Solaris on SPARC and x86 (Intel/AMD) hardware. Such differences will be annotated `on SPARC' or `on x86.'

This document uses the new binary-multiple prefixes described in the IEEE 1541-2002 standard. Prefixes like kilo, mega, giga, tera (k, M, G, T) refer to powers of ten; when the near-equivalent powers of two are meant, they are called kibi, mebi, gibi, tebi (ki, Mi, Gi, Ti). For example, a gigabyte (GB) is 10⁹ bytes, while a gibibyte (GiB) is 2³⁰ (1073741824). See http://physics.nist.gov/cuu/Units/binary.html for more details.

The programs described here have been tested on Solaris 7, 8, 9, and 10 on 32- and 64-bit SPARC systems, and on Solaris 10 on 32- and 64-bit x86 (IA32) systems. They are unlikely to function correctly on Solaris 2.6 or older without additional programming work. Solaris 9/x86 may work, but has not been tested. They are likely to work on Open Solaris, but since that is a moving target, no promises can be made.

Coraid EtherDrive® storage blades and Coraid SR RAID controllers have been tested. Any target device conforming to version 1 of the AoE protocol specification should work, provided the attached (or emulated) ATA disk supports logical-block addressing. Any ATA disk manufactured since 1995 ought to work fine.

The driver supports both 24 and 48-bit sector addresses, allowing disks as large as 128 pebibytes. Older versions of Solaris may be limited to 1TiB; see below for the full scoop.

Version 1.4 has these additional features and carries these warnings:

• Support has been added for Solaris/x86.
• DOS partition tables (fdisk labels) are recognized in a limited way, equivalent to that afforded by other Solaris drivers. See below for details.
• The normal Solaris tools for reading and writing disk labels (prtvtoc and format(1M)) may not work on an unlabelled disk larger than 1TiB. As a workaround, the AoE package includes a label-initializing program.
• Support for DOS labels may uncover new sorts of confusion in format (1M). In case of trouble, just use EFI labels unless a disk already has a VTOC label and cannot be rewritten from scratch.
• In early versions of Solaris 10, format and zpool may emit an odd but harmless message like
  Error occurred with device in use checking: No such device
  or
  warning: device in use checking failed: No such device
  when pointed at an AoE disk. This is probably a Solaris library quirk; in any case it is mended in the 11/06 release.
• ZFS on any sort of disk, not just just AoE, is surprisingly fragile in the face of I/O errors. Until Sun have repaired the problems, we suggest thinking twice before using ZFS.
• AoE may not be compatible with Sun Clusters; we haven't had a chance to test. Field reports are welcome.
• AoE may not work properly when run on an older, slower system through a 10Mbps le network interface. Using a 100Mbps hme network card or a faster CPU (any UltraSPARC should be OK) works fine.

A special broadcast message asks every target on the network, or the target with a specified address, to respond with a check-in message reporting its address and a few other parameters. When a target is first powered on (e.g. when an EtherDrive blade is inserted in a shelf) it broadcasts such a check-in response without being asked. Thus host software can find a target's Ethernet address given its target address, assemble a list of all targets on a network, and learn when a new target appears.

Pkgadd may warn that scripts are to be run as super-user. Installing the package runs add_drv(1M) to add three device drivers to the system, and uses awk to add entries to /etc/devlink.tab. (On Solaris 10, pkgadd claims /etc/devlink.tab is itself a set-userid file, apparently a garbling of the super-user script warning.) On an SMF system, installation runs svccfg(1M) to import the service manifest, and svcadm(1M) to work around a bug in establishing service dependencies.

Pkgadd may also warn that the ownership of the three /usr/share/man directories is being changed. The trouble is that these directories may not always exist, and there is no way both to tell pkgadd to create them if necessary and to tell it to leave existing permissions be.

To change from one version of AoE to another, remove the existing CORDaoe package and install the desired version as explained above. Remember to save /usr/kernel/drv/aoed.conf if you've customized it; otherwise it will be reset to its default contents when the new package is installed. If using ZFS, remember to zpool export and zpool import any pools already stored on AoE disks.

The package installs files in the following directories:

/usr/kernel/strmod

/usr/kernel/strmod/sparcv9

/usr/kernel/strmod/amd64

/usr/kernel/drv

/usr/kernel/drv/sparcv9

/usr/kernel/drv/amd64

Device driver binaries and configuration files. There are two device drivers and one STREAMS module, each in both 32- and 64-bit (sparcv9 or amd64) versions.

/usr/sbin

Configuration and control programs: aoectl, aoestart, aoestop, aoemkconf, aoelabinit, aoemon, aoeunlabel.

/etc/init.d

/etc/rc2.d

Non-SMF start-and-stop control script. /etc/init.d/aoe is installed on both SMF and non-SMF systems, the /etc/rc2.d/S00aoe link only on an non-SMF system.

/lib/svc/method

/var/svc/manifest/device

SMF start-and-stop control script /lib/svc/method/device-aoe and service manifest /var/svc/manifest/device/aoe.xml. Installed only on an SMF system.

/etc/aoe

Initially empty; holds one file per active aoechan, used (with fattach(3)) to hold the channel open.

/usr/share/man/man1m

/usr/share/man/man7d

/usr/share/man/man7m

Manual entries describing the programs and drivers.

/opt/CORDaoe/doc

Longer documents (this one, for instance) about the use and workings of AoE. See /opt/CORDaoe/doc/README for details.

/opt/CORDaoe/lib

Odds and ends used during package installation.

Any Ethernet device Solaris supports may be used. Special configuration may be required to enable jumbo frames; see Sun's documentation for details.

The Ethernet device need not be dedicated to AoE; in particular it may be shared with IP, though as noted above, this is often unwise.

To use more than one network device for AoE, declare additional channels as explained below.

Normally nothing special is needed in aoe.conf to permit jumbo frames, but jumbo support may need to be enabled in Solaris for the network device to be used. See the discussion of jumbo frames for the full scoop.

Once aoe.conf has been set up, AoE must be started:

• On an SMF system, AoE is disabled by default. To enable it, use svcadm(1M):
  svcadm enable svc:/device/aoe
  This both starts AoE right away and records that it should be started automatically on subsequent boots.
• On a non-SMF system, run
  /etc/init.d/aoe start
  or just reboot. Start-on-boot is enabled by default.

By default devices are created for targets 0/0/0 - 0/0/14 (i.e. cad0s* - cad14s*) and 0/1/0 - 0/1/14 (cad100s* - cad114s*): enough for two EtherDrive shelves or SR RAID devices numbered 0 and 1, each with up to fifteen blades. To make more devices requires editing a configuration file; in Solaris 9 and earlier, the system usually must be rebooted as well. See More devices for details.

In Solaris, names in /dev are symbolic links to real device files in /devices. For an AoE disk the real device name is of the form

Before an AoE disk device can be opened, it must be enabled. Normally this is done automatically: a target's check-in message is intercepted by aoemon; aoemon enables its devices, and reports to syslog that it did so.

aoectl list lists enabled targets. Aoectl may also be used to enable or disable specific targets, or to repeat the check-in broadcast so targets will repeat their responses. See the section on tools for details.

If a target isn't enabled but should be, try resetting it; e.g. pull the EtherDrive blade out of the shelf and push it back in, or put the SR lblade offline and then online. If all is well, within a few seconds aoemon should report that it has enabled that target.

Once enabled, AoE /dev/dsk and /dev/rdsk devices work like any other disks.

Format(1M) can be used to set up partition tables (the partition submenu) and to run simple read and read-write disk tests (analyze), modulo a few bugs and quirks described below. Low-level formatting, bad-block repair, and defect-list management (the format, repair, and defect menus) are not supported. A partition table may also be written with fmthard(1M) or read with prtvtoc(1M). On a brand-new huge drive, or one bearing a spurious old label, it may be necessary to use aoelabinit to initialize the label first.

If the disk doesn't have a label (partition table), the driver makes one up. In the made-up label, subdevice s0 (and s2 if the disk supports VTOC labels) is all of the disk intended for normal use, excluding parts reserved for use by diagnostics and the label itself.

Any file system except the root or /usr may be stored on an AoE disk. AoE is started very early in the boot process; AoE file systems listed in vfstab(4) and mounted automatically when the system boots. It is not possible to boot from an AoE device.

AoE disks may belong to ZFS storage pools, but beware the reliability problems (not specific to AoE) described below.

AoE disks may be used for swapping. AoE swap areas listed in vfstab will be added automatically. On a non-SMF system, swap areas are added twice, once before AoE has been started and once after. The first attempt produces a harmless message like

On an SMF system, if the devices/aoe service is enabled, it is automatically started if possible in single-user mode. To use AoE in single-user mode on a non-SMF system, ensure that /usr is mounted if it is a separate file system, then run /etc/init.d/aoe start.

If an unlabelled disk is larger than 1TiB, the disk driver makes no pretense. The whole-disk subdevice (cadnnn) accesses the entire physical device; none of the standard partition subdevices (cadnnnsn) may be used. Sun's standard tools report no label until one is written; that is the best that can be done within Sun's disk-driver interface. Format(1M) cannot handle an unlabelled disk large enough to require EFI labels; use aoelabinit as a workaround.

If a disk already has a VTOC label, everything works in the standard way. The eight standard partitions s0-s7 work as described in the label; Sun's standard tools display the label and can modify it.

If a disk already has an EFI label, and the system accessing it runs a version of Solaris new enough to have EFI support, everything also just works. The eight standard partitions s0-s7 are defined by the first eight EFI partitions. Sun's standard tools display the label and can modify it, with minor bugs.

If a disk already has an EFI label, pre-EFI Solaris can access it with limitations. The AoE driver reads the label and defines the partitions; the corresponding s0-s7 subdevices work; but Sun's standard tools don't understand the EFI label and report errors if asked to display it. In particular, format on a pre-EFI system doesn't work at all with an EFI-labelled disk. Aoeunlabel will wipe out an existing EFI label, allowing the disk to be partitioned from scratch.

Pre-EFI Solaris running in 32-bit mode can access at most 1TiB from a disk partition, even if the EFI label defines it to be larger. Pre-EFI Solaris in 64-bit mode can access the whole partition no matter how big. Pre-EFI Solaris cannot make a UFS file system larger than 1TiB, nor can such a system read a larger file system written by a newer system.

An AoE target is listed in format's disk-selection menu only if it is enabled. If an enabled device is unreachable (broken, removed, mistakenly enabled by hand), format may stall for a few seconds trying to access it, as it would for a configured-but-inaccessible SCSI disk.

Format exhibits a few bugs when working with an AoE disk larger than 1TiB:

• Format cannot work with an unlabelled AoE disk larger than 1TiB; it can't even write a label to such a disk. To work around this, use aoelabinit to write the first label.
• ATA devices report a device-model string. According to the ATA specification it is allowed to be 40 characters long. Prior to Solaris 10 11/06, format may mangle this string, truncating it to 24 characters and inserting a blank after the eighth.

User-mode programs responsible for starting AoE channels and for monitoring for unusual events (aoestart and aoemon, described in more detail below) log to syslog facility daemon. This log is where aoestart reports that an aoechan could not be started, and where aoemon reports that a target was automatically enabled. Aoemon also logs a packet dump when an error occurs: of the message received for an AoE-protocol or ATA error, of the message that couldn't be sent for a timeout.

Iostat(1M) lists AoE disks that have been used at least once since the last boot. Additional AoE-specific statistics may be fetched with kstat(1M).

The AoE subsystem comes with a default aoed.conf file declaring an initial set of targets. If more are needed, the file must be edited and the system told to reread it.

aoed.conf is a Solaris driver configuration file, in the form described in driver.conf(4). The easiest way to add to the file is to generate new entries with aoemkconf and edit them in. You may also write your own entries.

To summarize the general Solaris rules: a driver configuration file contains lines of text. Anything following # is a comment. Empty lines are ignored. Non-empty lines declare devices or driver properties (parameters, given as name=number or name="string"). Each line must be terminated with a semi-colon.

Each line in aoed.conf declares a device instance, specifying an instance number and target address:

name="aoed" parent="pseudo"

Constant values telling the system which driver this is and how it fits into the system.

instance=inst

Inst, a decimal number, is the instance number of this device, used to generate device names: instance=102 produces /devices/pseudo/aoed@102:a, /dev/dsk/cad102s0, and so on.

aoechan=cno

aoemaj=majno

aoemin=minno

Cno, majno, and minno are decimal numbers expressing the the target address: this is target cno/majno/minno.

These optional fields are allowed:

timeout=ms

Allow this device ms milliseconds before retrying or expiring a command. The default is 200ms. It may need to be lengthened for a long-latency device, or shortened to make the best of a badly-congested network, e.g. when running AoE over 10Mbps Ethernet (allowed but probably not a good idea).

hd=nhead

sec=nsec

Pretend this drive has cylinders of nhead heads and nsec sectors, ignoring any information returned by the hardware or the default numbers in the driver. Numbers stored in a valid disk label take precedence. Both hd and sec must be specified; if only one is supplied, it is ignored.

maxdata=size

Limit ATA data transfers for this device to at most size bytes. Usually needed only when using jumbo frames on problematic networks. Normally it is better to set the size for the whole aoechan rather than for individual targets.

maxbuf=ncmds

Allow at most ncmds unacknowledged commands for this target. Subsequent commands will be held until a pending command has been answered or has timed out. Normally initiator and target negotiate a suitable value, but it may occasionally be necessary to limit it on congested networks or when many initiators access a target concurrently.

The optional parameters must be added by hand; aoemkconf won't put them in.

The declarations in aoed.conf, and nothing else, connect instance numbers to target addresses. That the instance number is a decimal encoding of the target address is only a convention; nowhere in the driver subsystem is this assumed. If you write your own entries you may use whatever mapping you like, as long as no inst value is used for more than one device, and none is too large:² 20164 on a 32-bit SPARC system, 330382099 on 64-bit SPARC; 12483 and 204522252 on x86.

On Solaris 10, changes to aoed.conf can be made effective while the system is running:

On Solaris 9 and older systems, changes to aoed.conf have no effect until the next time the aoed driver module is loaded, and /devices and /dev are not updated until Solaris is told to do so. The simplest way to change the AoE configuration is to update aoed.conf and then perform a configuration reboot: reboot -- -r if Solaris is running, boot -r to the Open Boot ok prompt. It is also possible to make changes effective right away if no AoE device is in use, by unloading the aoed module and explicitly reconfiguring:

On Solaris 7, devfsadm may not exist (it was added by a patch) and in any case appears to be undocumented. The older equivalent, which works in either case, is:

It is wise to plan ahead, especially on a non-SMF system where the driver must be unloaded to reconfigure. For example, when installing a new EtherDrive shelf, add a device instance for each slot, not just for those you plan to use right away; when installing an EtherDrive Storage Appliance, create a few extra devices. To create hundreds of never-used devices would be silly, but the cost of a few extras is modest: a kibibyte or so inside the operating system per device, a handful of inodes for special files and symbolic links, extra entries in the /dev/dsk and /dev/rdsk directories.

If a target is removed while the system is running, it is prudent to disable it manually:

Removing a target's entry from /usr/kernel/drv/aoed.conf will save a little memory after the system is next booted. There's little point in doing this for a single target; it may make sense if many disks have been permanently removed.

When a target is removed from aoed.conf its entries in /devices or /dev may remain, even after a configuration reboot. Attempts to use such ghost devices return errors. If the same target is later restored to aoed.conf (and the driver reconfigured) the devices will work again. This is just the way some versions of Solaris work, especially older ones.

Changes to aoe.conf take effect when the system is next booted, or when svcadm refresh svc:/device/aoe (SMF system) or /etc/init.d/aoe restart (non-SMF) is run. Adding an aoechan usually requires adding entries to aoed.conf, so a non-SMF system usually must be booted anyway.

1. Enable jumbo frames in the appropriate Solaris device driver. See Sun's documentation for how to do this, and for the expected maximum frame size (usually expressed as MTU, the maximum Ethernet data payload). Make sure you set things up so that jumbo frames are automatically enabled at boot time for that device.
2. If there are any network switches between hosts and targets, find out the maximum frame size supported by each switch. If this is smaller than the jumbo size allowed by Solaris, configure the jumbo frame size by hand, as explained below.
3. If no switches are involved or all switches support sufficiently large frames, simply restart AoE or reboot with jumbo frames enabled in Solaris. The AoE subsystem will automatically detect the maximum frame size allowed by the Solaris network driver being used, reduce it if necessary to match that allowed by the AoE target, and use that size for subsequent I/O to that target.

The AoE subsystem can see the frame size configured in Solaris, and that available in the AoE target, but it can't see any limits imposed by switches. It's up to you to get that right.

A maximum frame size may be set for each AoE channel. This may be used only to reduce the frame size, not to increase it; the maximum size allowed by Solaris for the network device used by the channel cannot be exceeded. For each target accessed over this channel, the maximum size is further reduced if necessary to that supported by that target (discovered through part of the AoE protocol).

To specify a maximum frame size, add any of the following arguments to the channel's entry in /etc/aoe.conf:

mtu n

Frame data-payload size may be no greater than n bytes. If n is zero, the largest size supported by this channel's network device is used.

maxdata n

AoE ATA data transfers may be no larger than n bytes, rounded down to the nearest multiple of 512. maxdata n means the same as mtu n+22.

jumbo

nojumbo

Shorthand for mtu 0 (jumbo) or mtu 1500 (nojumbo).

The current maximum transfer size for a particular target is shown in the maxdata field of the conf kstat table.

Is bigger always better? Apparently not.

Our tests suggest that maxdata values of at least 3-4 kibibytes (6-8 sectors) produce large speed improvements over the default 1kiB (2 sectors); reads and writes through the file system are 2-3 times as fast. With even larger frames, the story is not as clear. For some operations speeds continue to increase, though not as markedly; for others there appears to be an optimal size (varying with type of operation and type of disk) after which speeds drop a little, though again not markedly.

It is not a bad idea just to use the largest value supported by your hardware and software; I/O speed will be much better than with standard 1500-byte frames, and the largest value will cause the least system overhead within Solaris. If it's important to squeeze every possible drop of speed out of your system, you may want to experiment to find the maxdata value that best matches your network, device drivers, disks, and application mix.

Those who do run their own tests are invited to share them with us, or to contact us for details about the simple tests we have used so far.

aoectl list address ...

List enabled targets at each address; if none given, all enabled targets.

aoectl enable address ...

aoectl disable address ...

Enable or disable targets at the addresses.

aoectl probe address ...

Broadcast a check-in request for each target address; if none given, for the wildcard address on every active aoechan. Any responding targets are enabled by aoemon.

For example:

aoectl list 0

List all enabled targets on aoechan 0.

aoectl probe 0/0/5 0/0/7 0//2

Probe for 0/0/5, 0/0/7, and any target on aoechan 0 with aoemin 2 no matter what its aoemaj.

aoectl disable 0/1

Disable all targets with aoechan 0, aoemaj 1.

aoectl list

List all enabled targets.

aoectl probe

Broadcast a wildcard check-in request to every active aoechan.

Aoectl need not be used during normal operation, only in exceptional cases or for monitoring. aoectl list helps spot targets that should be there but aren't; aoectl probe may be useful if the network is rearranged while the system is running; aoectl disable will prevent programs like format from stalling because a target has broken or been removed.

Aoestart starts a single aoechan according to its arguments. It is called automatically for each channel named in /etc/aoe.conf when AoE is started; normally it need not be run by hand. In fact each line of /etc/aoe.conf is simply used as arguments to aoestart.

Aoestop stops one, several, or all (-a) named aoechans. Normally there is no need to do this at all.

Aoelabinit is not meant to be a general-purpose disk labeller, just a workaround for the format bug forbidding access to large unlabelled disks.

Aoemon is started automatically when AoE is started. Only one copy is needed for all aoechans. If aoemon is not running, AoE disk devices will still work, but targets will not be automatically enabled and some errors may go unreported.

svcs svc:/device/aoe

Display the state of the service: online if all is well; disabled if not enabled; maintenance or offline if the service couldn't be started. In the latter case, log files /var/svc/log/device-aoe:default.log and /etc/svc/volatile/device-aoe:default.log may offer clues.

svcadm enable svc:/device/aoe

Start AoE if not already running: start aoemon, then call aoestart for each channel listed in aoe.conf. Remember that AoE should be started automatically when the system boots.

svcadm disable svc:/device/aoe

Abruptly stop AoE if it was running: call aoestop for each active channel, then kill aoemon. Remember that AoE should not be started on boot. All AoE disks become inaccessible, even if still mounted or open.

svcadm clear svc:/device/aoe

Reset the service from maintenance state, e.g. if it failed to start when last enabled.

svcadm refresh svc:/device/aoe

Shut down any currently-active AoE channels, and restart AoE with the channels now listed in /etc/aoe.conf. Equivalent to svcadm disable svc:/device/aoe; svcadm enable svc:/device/aoe, but quicker to type and less disruptive to active AoE devices.

Once enabled, the AoE service is started automatically early in the boot process. In particular the services that start swap devices and mount local file systems depend on svc:/device/aoe, and AoE is started even for the single-user milestone, though if AoE fails to start single-user mode still works.

Usually it is safe to abbreviate the service name to aoe.

On a non-SMF system, /etc/init.d/aoe is a conventional boot-and-shutdown shell script to start and shut down the AoE subsystem:

/etc/init.d/aoe start

Start AoE: start aoemon, call aoestart for each channel listed in aoe.conf.

/etc/init.d/aoe stop

Abruptly shut AoE down: call aoestop to stop all channels, then kill aoemon. All AoE disks become inaccessible, even if still mounted or open.

/etc/init.d/aoe restart

Shut down any currently-active AoE channels, start aoemon if it is not already running, then call aoestart for each channel listed in /etc/aoe.conf. Equivalent to aoe stop; aoe start but quicker to type and to execute, and less disruptive to active AoE devices.

With or without SMF, parts of the initialization script may be customized by placing shell-variable declarations in file /etc/default/aoe.options. Here are some of the things that may be set:

AOEBIN=dir

Where to find aoestart and aoemon; default /usr/sbin.

AOECONF=file

Where to find the file declaring channels; default /etc/aoe.conf.

AOESTARTOPTS="options"

AOEMONOPTS="options"

Additional options for aoestart or aoemon; default empty.

AOEDIR=dir

Directory for open-channel files; default /etc/aoe.

AOEDIRPERM=perms

Permissions with which to create $AOEDIR if it doesn't exist; default 700, i.e. only the owner (the super-user) may look within.

AOEPRESTART="shell-commands"

AOEPOSTSTART="shell-commands"

AOEPRESTOP="shell-commands"

AOEPOSTSTOP="shell-commands"

Additional shell commands to be executed just before or just after starting channels, or just before or just after stopping them. May be helpful if magic commands are required to initialize network interfaces.