A tour of the Solaris ATA-over-Ethernet (AoE) implementation

The driver calls ddi_create_minor_node(9F) with node type DDI_NT_PSEUDO, rather than the DDI_NT_BLOCK type normally used for disks. AoE-specific entries in /dev/devlink.tab create the /dev/dsk and /dev/rdsk links. This circumvents a bug in devfsadm(1M).

Every device instance must be listed in in driver configuration file aoed.conf, read by the system when the aoed driver module is loaded. Driver.conf(4D) gives the file format. Each entry must declare name="aoed", parent="pseudo", and these integer-valued properties:

instance=inst

Use instance number inst internally to identify this device to the driver; display it in device names and log messages.

aoechan=c

aoemaj=MM

aoemin=mm

Bind AoE target address c/MM/mm to this instance: c identifies an aoecomm channel, MM and mm give the AoE protocol major and minor addresses.

The copy of aoed.conf installed with the subsystem declares 30 devices: instances 0-14 and 100-114 for aoechan 0, aoemaj 0 and 1, aoemin 0-14. This is enough for two EtherDrive shelves, each with 15 slots. The system administrator may edit the file to add or remove devices (with aoemkconf or by hand) or to apply his own numbering convention, as explained in the Installation and Operation Guide.

By convention, decimal instance number instance=cMMmm corresponds to target address aoechan=c, aoemaj=MM, aoemin=mm. Whatever mapping appears in the entries in aoed.conf is honoured, however; the only program that knows about it is the aoemkconf program for generating configuration entries. Any desired instance-to-address mapping will work so long as instance numbers are not reused and none is too big.

In 32-bit versions of Solaris, 18 bits are available for minor device numbers. 13 (21 on x86) minor devices are created for each instance, allowing 262144/13 = 20164 (262144/21 = 12483) instances. In 64-bit systems, 32 bits are available, hence there may be 4294967296/13 = 330382099 (4294967296/21 = 204522252).

These optional integer-valued properties may be included:

timeout=ms

If an I/O operation hasn't completed within ms milliseconds, assume it never will; send the command again, or after several retries give up and report an error. The default is 200 milliseconds.

hd=nhead

sec=nsec

Assume this device has cylinders of nhead heads, and tracks of nsec sectors, ignoring any information supplied by the hardware or that invented by fudgegeom. Honoured only if both properties are included; if only one is given it is ignored.

maxdata=len

Allow a data segment of at most len bytes, overriding the default: the minimum of that offered in the target's Query-Config response message (1024 if no value returned), and that configured for the specified aoechan.

maxbuf=nbuf

Limit the maximum buffer count (number of concurrent messages the target can handle) to nbuf rather than the default 256. The value is in any case bounded above by that suggested by the target.

disksort=enable

If enable is 1, call the standard disksort(9F) kernel routine to sort queued disk operations; if zero, just use FIFO ordering.

On Solaris 9 and earlier versions, changes to aoed.conf take effect only when the aoed kernel module is loaded. If AoE disks have already been used, the file will not be effective until the driver is unloaded, which means every AoE disk device must be unmounted and closed. Often it is simplest just to reboot the system.

On Solaris 10, update_drv(1M) updates the configuration. With option -f it can do so without a module unload. Device instances presently in use cannot be changed on the fly, but new devices may be added without rebooting.

The aoed driver source code is broken into several files:

aoed.c

Main program: official driver entry points and data structures.

addadk.c

aduscsi.c

Handle ATA- and SCSI- (sic) specific ioctl calls.

adio.c

Generate and send AoE commands for an I/O request, or as needed during device initialization. Process AoE replies. Retransmit commands whose answers don't arrive in a specified time; cancel commands that have suffered too many retries.

addos.c

adefi.c

advtoc.c

Read and write disk labels and partition tables; separate implementations for each of the several label formats.

adcmd.c

Allocate, free, and manage lists of pending AoE commands.

adsubr.c

Miscellaneous subroutines: find the Adrive structure associated with a Solaris device; error-message helpers.

adtrace.c

Simple event-tracing mechanism; used only when debugging the driver.

aoed.h

Header file declaring data types and instances and procedure prototypes used in more than one source file.

dadkio32.h

Header file declaring a 32-bit version of certain data structures used by the Solaris DKIOCTL_RWCMD ioctl. For some reason Sun's header files leave these out.

Many subroutines and data objects are meant to be shared among aoed source files but not with the rest of the kernel. The system provides no known way to do exactly that. (Perhaps ld(1) has an option to do the trick, but we have yet to discover it.) If every such symbol began with a prefix unique to aoed all would be well, but that would clutter the source code. As a compromise, such names are overlaid with macros in aoed.h, in effect prepending aoed_. For example, the prototype for acalloc is preceded by

To make it easier to include optional code only as needed (trace in particular), all object files except main program aoed.o are collected into object library adlib.a. The module binary is generated by running

None is used outside the aoed module.

• An integer state with one of the following values:

  ADCLOSED

  No process has this device open; I/O forbidden.

  ADWAOE

  First stage in device setup: waiting for AoE Query-Config response.

  ADWATA

  Second stage in device setup: Query-Config response received, awaiting ATA IDENTIFY response. MAC address, max-buffer, and max-data-segment values valid.

  ADGATA

  Third stage in device setup: ATA IDENTIFY response received, number of sectors valid. I/O is now allowed, but only to the wd partition.

  ADWPTAB

  Fourth stage in device setup: waiting for disk label to be read.

  ADREADY

  Setup complete; regular I/O allowed to any partition defined by the label, whether real or synthetic. Disk label, geometry, and partition table valid, as allowed by the label type.

  State values are ordered as above: for example, ap->state >= ADGATA is true only if IDENTIFY information has been received.
• A drive-is-closing flag, used to ensure that no new I/O is begun during adclose.
• Bit-masks (of integral type Pbits to hide the number of bits required) indicating which block and which character minor devices are open; a count of the outstanding layered opens. Used to decide whether this is the first open or last close.
• Static configuration values:
  • Target address (aoechan, aoemaj, aoemin) of the associated AoE device; determined from properties initialized from aoed.conf.
  • Optional property values from aoed.conf: I/O timeout value, substitute head and sector counts.
  • Solaris instance number and dev_info_t pointer.
  • Solaris major device number, discovered when the device is first opened; needed for a few kernel service routines that require a complete dev_t as argument.
  • Other static flags: maximum data-segment size was set by an configuration property and should not be overridden; this system uses the older, incompatible argument to the DKIOCGETEFI and DKIOCSETEFI ioctls .
• Configuration properties of the AoE target:
  • Target's Ethernet MAC address.
  • Maximum buffer count, i.e. number of concurrent messages the target can handle.
  • Maximum data-segment size, i.e. the longest read or write transfer allowed in an ATA command.
• Geometry, partition, and other information about the target disk:
  • Number of logical blocks in the target disk.
  • Dynamic flags, including the label type: AVLABVTOC, AVLABEFI, AVLABDOS, or (AVLABDOS|AVLABVTOC) (when the VTOC label is encapsulated in a DOS partition).
  • Cylinder size; often a fake value invented by fudgegeom or adreceive.
  • Solaris dk_geom structure describing disk geometry; meaningful only with label type AVLABDOS or AVLABVTOC.
  • Solaris vtoc (volume table-of-contents, i.e. on-disk label; valid only under label type LABVTOC.
  • Starting sector and length of each partition, including the whole-disk partition. An invalid partition has length zero.
  • Starting sector and length of the Solaris-labelled disk area. If the disk has a DOS partition encapsulating a Solaris label, these are the starting sector and length of that partition; otherwise zero and the length of the disk.
  • Serial-number, disk firmware version, and disk model strings returned by the ATA IDENTIFY command.
• Information about pending and active I/O requests:
  • List of Acmd structures available for allocation by acalloc.
  • Tag value to be used in the next command sent to this target.
  • Count of active commands, those in the list of pending commands in this channel's Achan.
  • Solaris diskhd structure containing a queue of buf structures for pending I/O requests, suitable for use by disksort(9F).
  • Pointers to the head and tail of the list of active buf structures for this device, i.e. those whose transfers have been at least partly begun.
• A kmutex_t lock to prevent concurrent access to non-static values in this Adrive.
• A kcondvar_t condition variable used while waiting for initialization to finish.
• Structures used to collect statistics displayed by iostat and kstat(1M): a kstat_t of type KSTAT_TYPE_IO for the whole drive, another for each partition (including the whole-disk partition), one named aoedinst,stats for a handful of per-drive counters specific to aoed, another named aoedinst,conf to display per-drive configuration values.

Each Adrive has a pool of Acmd structures, allocated when the device is first opened, freed on last close. The pool is initialized with as many Acmds as the target allows concurrent commands. When a command is composed and sent, an Acmd is taken from the pool; while the command awaits a response, the Acmd is kept in a per-channel pending-command list; when a response is received or the command is abandoned after a timeout, the Acmd is returned to the pool. If the pool is empty no new commands may be sent until an outstanding command has completed or timed out and its Acmd has been returned to the pool.

Each Acmd contains:

• The AoE command message to be sent, excluding any data sectors to be written.
• A pointer to the associated Adrive.
• A pointer to the associated buf structure. If NULL this is not an I/O command.
• A pointer to the associated kcondvar_t condition variable; NULL if none.
• The AoE tag value for this command.
• How long to wait for a response before declaring that this command has timed out; how many retries have occurred, and how many are allowed.
• When this command was first started, and when it next times out, in absolute clock ticks as returned by ddi_get_lbolt(9F).
• If an I/O command, the absolute sector number of the first sector of this transfer, and the byte offset and length of the segment handled by this command.
• Next-Acmd and previous-Acmd pointers, for the several lists in which Acmds may be stored.
• A state value, used only for sanity checks: value CSFREE if this Acmd is in the drive's free-command pool; CSCHAN if on the channel's active-command list; CSLOOSE otherwise. If an Acmd in the pool or on the list is found to be in the wrong state, or if an Acmd being added to either is not in state CSLOOSE, the system panics.

• Pointers to the head and tail of a list of Acmds representing commands awaiting responses from this aoechan.
• A timeout_id_t identifying this channel's instance of chantimer, if any.
• A channel-is-active flag.
• A kmutex_t lock preventing concurrent access to the Achan.

Every buf structure for a pending or active I/O transfer has an associated Abuf structure, allocated when the buf is accepted by adstrategy. The address of the Abuf is stored in bp->b_private. The Abuf is freed before biodone(9F) is called.

Macro bptoabuf(bp) is a shorthand for (Abuf *)(bp->b_private).

Each Abuf contains:

• The number of AoE commands outstanding for this buf.
• The absolute logical sector number to be used in the next AoE command generated for this transfer.
• The offset within the transfer to the first byte to be read or written by the next command.
• The total number of bytes to be transferred.

Standard Solaris loadable-module and device-driver entry points and data structures are supplied: in particular _init, _fini, and _info routines, a modlinkage structure (and its many children), dev_ops and cb_ops structures. A _depends_on string declares a dependency on aoecomm, since aoed calls procedures supplied by that module.

Only _init, _fini, and _info are global. _init calls mod_install to tell the system where to find the modlinkage structure through which the other data structures and routines can be located. Apparently _depends_on need not be global to work.

Internal routine adreceive is registered with aoecomm as the message receiver for each aoechan that has been used to send at least one message. Internal routine chantimer runs as a self-renewing timer, with one instance per aoechan.

Attach routine adattach works as follows:

• Allocate an Adrive for this device instance. Initialize it with constants at hand: instance number and dev_info_t pointer; aoechan, aoemaj, aoemin, other aoed.conf-entry properties, fetched with ddi_get_prop_int(9F). If any required property is missing, return failure.
• Call solversion; if this system is earlier than Solaris 10, set the flag indicating an old-style argument to the DKIOCGETEFI and DKIOCSETEFI ioctls.
• Call ddi_create_minor_node(9F) to create each of the eighteen minor devices for this instance.
• Initialize locks and condition variables in this Adrive.
• Call ddi_report_dev(9F) to inform the system that the instance is ready.

Open routine adopen works as follows:

• Call aoecomm_devenab to check that the AoE target has been enabled. If not, immediately return ENXIO.
• Create kstat_t structures for the drive and for the partition being opened, as necessary.
• If this is a non-blocking open (FNDELAY or FNONBLOCK set in open flags) and the Adrive state is ADCLOSED, call driveinit to start the initialization process.
• If neither FNDELAY nor FNONBLOCK was set, call openwait to start initialization if necessary and wait for completion. If openwait fails, return the error reported. Check whether the partition being opened is of length zero; if so, return ENXIO.
• Mark this partition open according to the type of open call: for block or character open, set the appropriate bit in the appropriate open-subdebvice bitmask; for a layered open, increment the layered-open count.

Drive initialization comprises these steps:

1. Driveinit called; drive state is ADCLOSED

Allocate a single Acmd for this drive; broadcast an AoE Query-Config command with the desired aoemaj and aoemin addresses to the desired AoE channel, to locate the target. Set drive state to ADWAOE.

2. Query-Config response arrives matching the pending Acmd; drive state is ADWAOE

Store the maxbuf value reported in the Query-Config response (limited by the maximum configured for this drive), and the MAC address whence it came. Call acinit to top up this drive's Acmd pool to maxbuf entries. If the maximum data-segment size wasn't specified as a configuration property, extract that in the response message, interpreting zero as the default value 1024; compare it with the value configured for the channel, returned by aoecomm_maxdata; store the lesser in ep. Compose and send an ATA IDENTIFY command. Set state to ADWATA.

3. ATA response message arrives; drive state is ADWATA

Extract size and geometry information and device-type strings from the presumed IDENTIFY response. If the disk is VTOC-proper, initialize the dk_geom structure in ep and call fudgegeom to tweak the numbers if necessary; if VTOC-improper, invent a cylinder size such that the largest possible cylinder number (given the size of this drive) is bounded by ULONG_MAX, for disksort(9F). Set up whole-disk partition limits. Fill in the configuration kstat_t structure, now that dynamically-determined values like maxdata and mibsize are known. Set state to ADGATA. Awaken any processes sleeping in openwait for this drive.

4. Drive state ADGATA discovered within openwait

Compose a device ID object (ddi_devid_register(9F)) and several string properties; the former may be used by some Solaris programs, the latter are an invention and may disappear in the future. Set state to ADWPTAB. Call readptab to read the disk label, filling in all remaining partition entries, the label type, and possibly other label-specific data in the Adrive. When all is well, set state to ADREADY.

Locks are used to assure that at most one process at a time is working on initializing a given drive. Other processes desiring to access the drive sleep until initialization succeeds or is abandoned due to an error.

• If a block or character close, clear the open-subdevices bit for the appropriate partition and type; if a layered-open close, decrement the counter.
• If at least one partition for this drive remains open or there is at least one outstanding layered open, return success now.
• Set the drive-is-closing flag. Loop, sleeping if necessary on the Adrive's condition variable, until both the active- and pending-buf lists have drained and the active-command count is zero.
• Call acclear to free the Acmd pool. Clear the drive-is-closing flag. Set drive state to ADCLOSED.

Detach routine addetach works as follows.

• Discard all minor devices for all device instances (ddi_remove_minor_node(9F) with second argument NULL).
• If the list of active buf structs for this drive is not empty, log an error message and return failure. (This shouldn't happen.)
• Discard any remaining Acmds for this drive; clear out the Acmd pool.
• Free any kstat_t structures associated with this drive, including those for all partitions.
• Destroy the kmutex_t locks and the kcondvar_t condition variable.
• Free the Adrive.

The system calls addetach only after all references to this device instance have been closed; adclose blocks until no pending I/O operations remain. Hence adddetach need show no mercy to any I/O still unfinished.

All I/O transfers are therefore queued by strategy routine adstrategy, which works as follows:

• If the drive-is-closing flag is set for this drive, abort the transfer with ENXIO.
• Check that this is a reasonable transfer; in particular that the length is a multiple of DEV_BSIZE (512). If not, abort the transfer with ENXIO.
• If the drive state is < ADGATA, call openwait to wait until I/O is possible; evidently a non-blocking open is in effect. If openwait reports an error, abort the transfer with that error code. (Why ADGATA rather than ADREADY? So the normal I/O path may be used to read the disk label.)
• Check that the requested transfer is within the current partition:
  • If it begins after the end of the partition, abort with ENXIO.
  • If it begins exactly at the end of the partition, set bp->b_resid set to bp->b_bcount (so read will return zero), call biodone(9F), and return.
  • If it spans the end, shorten bp->b_bcount to fit.
• Allocate an Abuf for this buf structure; store its address in bp->b_private. Initialize the count and absolute starting block number.
• Store the cylinder number in which this transfer begins in bp->b_resid; call disksort(9F) to insert this transfer in the drive's queue of pending requests.
• Call adstart to start I/O if possible.

Ioctl routine adioctl calls openwait to check that initialization is complete (in case this was a non-blocking open), then switches on the ioctl command code.

The data-model rules for ioctl in Solaris complicate matters. When a 32-bit program makes ioctl calls under a 64-bit kernel, conversions may be necessary, depending on the data used by the particular ioctl command. Sometimes the 32- and 64-bit forms of a data structure have the same size and layout; sometimes they differ, and the appropriate version must be filled in and returned. In one case they differ but Sun doesn't ship a header file declaring the 32-bit form even though it is needed for format(1M). See Writing Device Drivers for more about this mess.

In another case, handled by efiioc, the argument format differs according to the Solaris version. This is why adattach checks the version and sets a flag.

The following standard disk-device commands are supported. Many, but not all, are listed in the Writing Device Drivers book and described in somewhat more detail in dkio(7D).

DKIOCINFO

Fetch information about the disk controller. Returns constant controller type DKC_DIRECT, with a constant large controller number (in the hope that it won't conflict with a real one), the instance number as the disk unit number, slave number zero.

DKIOCGAPART

DKIOCSAPART

Fetch or set the starting block number and length of all eight standard Solaris partitions on this drive. Only the in-core Adrive is written, not the disk label.

DKIOCPARTINFO

Fetch the starting block number and length of the partition open on this file descriptor.

DKIOCGGEOM

DKIOCG_PHYGEOM

Return the drive's current dk_geom structure.

DKIOCSGEOM

Update the dk_geom structure in ep, without changing the on-disk label.

DKIOCGMEDIAINFO

Return a dk_minfo structure containing the media type (always DK_FIXED_DISK), sector size (always DEV_BSIZE), and total sector count of the disk.

DKIOCREMOVABLE

Is this a removable-media device? Always returns `no.'

DKIOCSETEFI

DKIOCSETEFI

DKIOCPARTITION

EFI-label-specific calls handled by efiioc.

DKIOCGVTOC

DKIOCSVTOC

VTOC-label-specific calls handled by vtocioc.

DKIOCGMBOOT

DKIOCSMBOOT

DOS-label-specific calls handled by dosioc.

DIOCTL_RWCMD

DIOCTL_GETMODEL

DIOCTL_GETSERIAL

ATA-specific calls handled by dadkioctl.

USCSICMD

SCSI-specific (sic) call handled by aduscsi.

To send an AoE command from within the driver:

• Call acalloc to allocate an Acmd from the pool for this drive. If none is available, no more commands may be sent for now. Acalloc initializes the Adrive pointer, header-length field, and the common part of the AoE header (AoE command, aoemaj and aoemin, version code and other flags, unique tag value), and zeroes all other fields.
• Fill in the expiry interval and maximum retry count. Fill in the remainder of the AoE command, e.g. ATA register values for an ATA command, but not data sectors to be written. If this is a transfer, set the buf pointer, the starting block number, and the transfer length and offset.
• Call adstcmd to send the command and place the Acmd on the pending-command list for this aoechan.

Chantimer is called at regular intervals to discover commands that must be retransmitted or expired.

AoE messages arrive by calls to adreceive, which calls adlookup to locate the corresponding Acmd, then processes the message according to its command code. If the message is garbled or otherwise questionable, the Acmd is returned to the channel's pending-command list; otherwise it is freed to the pool for its Adrive. A garbled, erroneous, or unsolicited (no Acmd) message is logged and discarded.

Here is a summary of major internal procedures, grouped by source file. Unless otherwise noted, each is global within the aoed module, but is not meant to be called from elsewhere in the system; hence aoed.h supplies both a prototype and a name-hiding macro, as explained above.

Return zero when ep has reached state ADREADY, blocking if necessary; or return a nonzero errno value if initialization fails.

If ep is in state ADCLOSED (initialization hasn't started yet), call driveinit to get things started. At each subsequent step prior to ADREADY, call cv_wait_sig(9S) to block on the condition variable in ep. If cv_wait_sig is interrupted by a UNIX¹ signal, return EINTR.

Openwait is local to aoed.c.

1. If the drive state is < ADGATA or the disk size is not known, return -1.
2. If fetchvtoc finds a proper VTOC label, return 1, leaving the label-type flags set to AVLABVTOC.
3. If fetchefi finds a proper EFI label, return 1, leaving the label-type flags set to AVLABEFI.
4. If fetchdos finds a proper DOS label:
   • If a DOS partition is marked as possibly containing an encapsulated VTOC, and this is a Solaris/x86 system, call fetchvtoc . If no proper VTOC label is found, call fakevtoc to invent one. In either case return 1, leaving the label-type flags set to AVFLABDOS|AVLABVTOC.
   • If no DOS partition has the encapsulated-VTOC type, or in any case on Solaris/SPARC, return 1, leaving the label flags set to AVLABDOS.
5. If fakevtoc can make a fake VTOC label, return 0, leaving the label-type flags set to AVLABVTOC.
6. Otherwise return 1, with no label-type flags set.

Readptab is local to aoed.c.

Write zeroes to absolute sector bno on device ep; return zero for success, nonzero errno value on failure. Calls adrwkern to do the real work.

Adrwkern makes a synthetic buffer header and calls adstrategy; adrwuser makes a struct uio and calls physio(9F). Both operate on the whole-disk partition.

DIOCTL_RWCMD

Read from or write to an absolute address on an ATA disk, regardless of the partition table. Uses adrwuser to do the dirty work. .

DIOCTL_GETMODEL

DIOCTL_GETSERIAL

Return the ATA device-model and serial-number strings, learned from an ATA INQUIRY command and stored in the Adrive during device initialization.

Aduscsi is needed to work around bugs in some versions of format(1M).

Those familiar with SCSI may note that the SCSI standard doesn't define READ BLOCK LIMITS for disk devices. Format calls it anyway.

The ATA IDENTIFY command returns a device-model string; SCSI INQUIRY returns separate vendor and model strings. The two SCSI strings together afford less than half the maximum length of the single ATA string, and in fact some ATA devices return strings too long to fit in the SCSI message. The faking code does the best it can.

Examine the boot sector (sector 0) of drive ep for a valid DOS partition table. If none is found, return 0. Otherwise scan the four direct partitions for one of type SYSID_SUNOS or SYSID_SUNOS2. If such a partition is found, set the encapsulated-label partition start and length to those of the partition, and record fake geometry parameters in ep. If there is more than one such partition, use the highest-numbered. Whether an encapsulated-label partition was found or not, set the AVLABDOS label-type flag and return 1.

DKIOCGMBOOT

DKIOCSMBOOT

Call adrwkern to read or write the boot sector (where the DOS label lives) on the disk, to or from the buffer pointed to by arg.

Search drive ep for a valid EFI label, trying the primary (sector 1) and backup (last sector) GPT locations as necessary. If a label is found, store the starting sector address and length of the first eight partitions in the drive's partition table, set label-type flag to AVLABEFI, and return 1. If there is no label, return 0; if an I/O error occurred, return -1.

Invalidate any existing EFI label on the drive by zeroing the primary and backup GPT sectors; clear label-type flag AVLABEFI.

DKIOCGETEFI

DKIOCSETEFI

Read or write absolute sectors of the drive, as specified by the user-space dk_efi_t structure addressed by arg, via calls to adrwuser. Take care to account for the different format the structure had in Solaris 9. For DKIOCSETEFI only, call zaplabels with flag argument AVLABEFI, then set that flag in the label-type flags of ep. The Solaris libefi(3LIB) library uses these calls; the real label processing is done by the library, not the kernel.

DKIOCPARTITION

Read the EFI partition descriptor for the partition specified in the partition64 structure addressed by arg directly from the disk (not from the driver's in-core partition table). Copy the partition's starting sector, length, and UUID to the partition64 and copy it back to the user.

Search the Solaris-labelled part of the disk for a valid VTOC label, checking the primary location (relative sector 0) and the backups (sectors 1, 3, 5, 7, and 9 in the last track) as necessary. If a valid label is found, use its contents to fill in the drive's vtoc and dk_geom structures and its partition table; set the drive's label type to LABVTOC; and return 1. If no label is found or an I/O error occurs, return -1.

DKIOCGVTOC

Return the vtoc structure stored in ep.

DKIOCSVTOC

Update the drive's partition table from the vtoc structure addressed by arg; copy that vtoc to ep; then, using that vtoc and the drive's current dk_geom, update the primary copy and all backups of the on-disk VTOC label. Call zaplabels with argument type AVLABVTOC.

1. If ndelay is nonzero (a non-blocking open is in progress), reset the MAC address for this drive to the global broadcast address and return success. The real work will be done when driveinit is called by a subsequent openwait call.
2. Confirm that the target is enabled, and that the target channel is active.
3. Register adreceive as the receiver for the target channel.
4. Start chantimer for this channel if necessary.
5. Call acalloc to allocate a single Acmd, and immediately use it to send an AoE Query-Config command.

Initially ep should be in state ADCLOSED. If all is well, return 1 with ep in state ADWAOE; if an error occurs, return -1 with the state unchanged.

Acp really points to an Achan, but the argument type must be void * to satisfy the rules of timeout(9F).

Examine each Acmd awaiting a response from the aoechan associated with ap. If an Acmd has expired but another retry is allowed, retransmit it. If no more retries are allowed, cancel the command:

• If there is an associated buf structure, pass it to biodone(9F) with error ETIME, and call adkillbuf to free any other Acmds associated with the same buf.
• If there is an associated condition variable, call cv_broadcast(9F) to awaken processes waiting on it.

A copy of chantimer normally runs every TMO_CLOCK ticks (20 ms) for each AoE channel that has been used at least once by the aoed module. If a command was retransmitted, chantimer quits, and the next run begins in TMO_CLOCKHOLD ticks (10 ms), to avoid flooding a congested network with retries.

Chantimer is local to adio.c, though it is called from outside the module via timeout.

• If mp is NULL, the channel is shutting down. Locate the corresponding Achan, clear the channel-is-active flag, and return.
• If the message is obviously malformed (too short, invalid version, not an AoE response), call aoecomm_log to log it with code ACLILLFORM; return.
• Call adlookup to locate the Acmd for which this is a response, and to remove it from the active-command list. If none is found, log it with code ACLUNSOL and return.
• If the message is a well-formed Query-Config response and the drive state is ADWAOE, copy the MAC address and maxbuf value to the Adrive; adjust the maximum data-segment size if required; call acfree to free the Acmd; and move to the next initialization step.
• If the message is a well-formed ATA-command response:
  • In drive state ADWATA, this is the response to an ATA IDENTIFY command: store data; free the Acmd; and move to the next initialization step.
  • In drive state ADGATA, ADWPTAB, or ADREADY, this is the response to an ATA READ or WRITE request. Locate the corresponding buf structure; free the Acmd. Store read data if any; if the transfer is now complete or an error occurred, free the Abuf and call biodone(9F). If the device-is-closing flag is set and both the active-transfer list and the pending-transfer queue are empty, signal the condition variable to awaken adclose. In any case call adstart to start more I/O if any remains pending, whether for this transfer or another another.
• If the message is ill-formed or the drive in an unexpected state, log the message with code ACLILLFORM or ACLUNSOL, and return the Acmd to the active-command list.

Adreceive is local to adio.c, but is usually registered with aoecomm as the receiver for one or more channels.

Start I/O transfers on drive ep, looping until no further Acmd is available (acalloc returns NULL) or no buf structure remains queued.

Depending on the transfer length and the maximum data-segment size for this drive, several commands may be required for a single buf structure. Adstart starts as much as it can. If it was forced to stop (ran out of Acmds) partway through a buf, the next call to adstart continues with that buffer before starting another.

Compose the AoE command described in dp in a new STREAMS buffer, and call aoecomm_send to send it. On success, call adstore to add dp to the channel's active-command list and return 1. On error, return -1: unreasonable dp contents, aoecomm_send failed.

Before calling aoecomm_send adstcmd locates the Achan for the target channel and checks the channel-is-active flag. If the flag is clear, aoecomm_initdriver is called to re-register adreceive as the receiver for this channel; if aoecomm_initdriver succeeds, the flag is then set. This affords recovery when a channel is shut down and then restarted: adreceive clears the flag when aoecomm reports the shutdown; adstcmd re-registers the receiver when next possible, and sets the flag when it happens. Pending messages presumably time out and are retried. If the channel comes back within the timeout, no data are lost; if not, an I/O error is reported as for any other timeout.

These fields must have been filled in in dp:

dp->ep

Adrive to which the command should be sent. Both the target address (aoechan, aoemaj, aoemin) and Ethernet MAC address are significant; the latter may be the broadcast address.

dp->tag

Tag value for this command.

dp->aoehd

dp->aoelen

AoE command to be sent, excluding data to be written. aoehd is preallocated, with room for AOEHEADLEN (32) bytes, of which aoelen are used by this command.

dp->exptime

Minimum time, in clock ticks, allowed before declaring that this command has expired. Zero means never.

dp->maxtries

dp->ntries

This command may be sent at most maxtries times, i.e. at most maxtries-1 retries; it has already been sent ntries times.

dp->bp

dp->len

dp->off

If bp is not NULL and B_READ is not set in its flags, append len bytes of data from that buffer, starting at offset off within the buffer.

Dp->expires is set to the current time in clock ticks (read from ddi_get_lbolt(9F)) plus the expiry interval, with the latter lengthened a little for retries:

Extract the tag value and target-address numbers from the AoE message pointed to by p. Search the active-command list for channel chan for an Acmd with a matching tag, associated with an Adrive with a matching target address. If a match is found, remove the Acmd from the active-command list, set its state to CSLOOSE, decrement the active-command count in the Adrive, and return the address of the Acmd. If there is no match, return NULL.

Add dp to the active-command list for channel chan; increment the active-command count in the corresponding Adrive. The state of dp must be CSLOOSE; it is set to CSCHAN.

If dp is on the active-command list for channel chan or cp, remove it; set its state is set to CSLOOSE; decrement the active-command count of the corresponding Adrive.

Cp must be locked before calling adcremove.

For each Acmd on the active-command list for channel chan and associated with ep: remove and free (acfree) the Acmd; decrement the active-command counter for ep.

For each Acmd on the active-command list for channel chan and associated with bp: remove and free (acfree) the Acmd; decrement the active-command counter for the corresponding Adrive.

Allocate an Acmd from the pool associated with ep, and initialize its contents as follows:

• Set the Adrive pointer to ep.
• Store ep->nexttag & ~TAGOFFSET in the tag field of the Acmd, taking care to skip the value zero (used by AoE targets for unsolicited broadcast replies). Increment ep->nexttag for next time.
• Set the AoE command length to hdlen. Fill in the common header: standard protocol-version and flag values, command type aoecmd, aoemaj and aoemin from ep, tag from that just invented.
• Set the state to CSLOOSE.
• Zero all other fields.

Messages with tag values greater than or equal to TAGOFFSET (0x80000000) are reserved for non-kernel diagnostic programs. There are none such yet in the Solaris implementation. Originally the sense was inverted, with TAGOFFSET set in messages generated by the Solaris kernel driver, lesser values reserved for non-kernel use; this was changed in version 1.3.3 for consistency with the Linux AoE implementation.

Return dp to the pool associated with ep, setting its state to CSFREE.

Allocate new Acmds from system memory, initialize them with state CSFREE, and add them to the pool associated with ep until the pool contains at least nfree entries or the system runs out of memory.

If the pool is not empty, but has fewer than nfree entries, it is topped up to nfree. If it is already larger than nfree the system panics.

Return a pointer to the Adrive corresponding to Solaris device dev. If none exists, log an error message (including s if non-NULL) and return NULL.

Either routine stores the string in buf, truncating safely after len bytes if necessary. The return value is buf.

The message will be truncated safely at 300 characters.

Return a short string name for partition part, in the style used for names in /devices: a for part 0, b for 1, and so on to h for 7; wd for 8; ??? for anything else.

• If (f&AVLABDOS) == 0, call zapdos.
• If (f&AVLABVTOC) == 0, call zapvtoc.
• If (f&AVLABEFI) == 0, call zapefi.

Solversion works by examining global value utsname.release. This is admittedly a hack, as is the very notion of this routine, which should be used only as a last resort.

8. User-mode component details

8.1. Aoestart

8.2. Aoestop

8.3. Aoectl

8.4. Aoemon

8.5. Library routines

8.5.2. achan.c

8.5.3. trdwr.c

8.5.5. debug.c

8.6. Aoelabinit

8.7. Aoeunlabel

8.8. Aoemkconf

8.9. /etc/init.d/aoe, /lib/svc/method/device-aoe