Devices ibd(7D)
NAME
ibd - Infiniband IPoIB device driver
SYNOPSIS
/dev/ibd*
DESCRIPTION
The ibd driver implements the IETF IP over Infiniband proto-
col and provides IPoIB service for all IBA ports present in
the system.
The ibd driver is a multi-threaded, loadable, clonable,
STREAMS hardware driver supporting the connectionless Data
Link Provider Interface, dlpi(7P)). The ibd driver provides
basic support for the IBA Unreliable Datagram Queue Pair
hardware. Functions include QP initialization, frame
transmit and receive, multicast and promiscuous mode sup-
port, and statistics reporting.
Use the cloning, character-special device /dev/ibd to access
all ibd devices installed within the system.
The ibd driver is dependent on GLD, a loadable kernel module
that provides the ibd driver with the DLPI and STREAMS func-
tionality required of a LAN driver. Except as noted in the
Application Programming Interface section of this manpage,
see gld(7D) for more details on the primitives supported by
the driver. The GLD module is located at
/kernel/misc/sparcv9/gld on 64 bit systems and at
/kernel/misc/gld on 32 bit systems.
The ibd driver expects certain configuration of the IBA
fabric prior to operation (which also implies the SM must be
active and managing the fabric). Specifically, the IBA mul-
ticast group representing the IPv4 limited broadcast address
255.255.255.255 (also defined as broadcast-GID in IETF docu-
ments) must be created prior to initializing the device. IBA
properties (including mtu, qkey and sl) of this group is
used by the driver to create any other IBA multicast group
as instructed by higher level (IP) software. The driver
probes for the existance of this broadcast-GID during
attach(9E).
APLICATION PROGRAMING INTERFACE (DLPI)
The values returned by the driver in the DLINFOACK primi-
tive in response to your DLINFOREQ are:
SunOS 5.11 Last change: 20 Jul 2007 1
Devices ibd(7D)
o Maximum SDU is the MTU associated with the
broadcast-GID group, less the 4 byte IPoIB header.
o Minimum SDU is 0.
o dlsap address length is 22.
o MAC type is DLIB.
o The sap length value is -2, meaning the physical
address component is followed immediately by a 2-
byte sap component within the DLSAP address.
o Broadcast address value is the MAC address consist-
ing of the 4 bytes of QPN 00:F:F:F prepended to
the IBA multicast address of the broadcast-GID.
Due to the nature of link address definition for
IPoIB, the DLSETPHYSADREQ DLPI primitive is
not supported.
In the transmit case for streams that have been put
in raw mode via the DLIOCRAW ioctl, the DLPI appli-
cation must prepend the 20 byte IPoIB destination
address to the data it wants to transmit over-the-
wire. In the receive case, applications receive the
IP/ARP datagram along with the IETF defined 4 byte
header.
WARNING
This section describes warning messages that might be gen-
erated by the driver. Please note that while the format of
these messages may be modified in future versions, the same
general information will be provided.
While joining IBA multicast groups corresponding to IP mul-
ticast groups as part of multicast promiscuous operations as
required by IP multicast routers, or as part of running
snoop(1M), it is possible that joins to some multicast
groups can fail due to inherent resource constraints in the
IBA components. In such cases, warning message similar to
the following appear in the system log, indicating the
interface on which the failure occurred:
NOTICE: ibd0: Could not get list of IBA multicast groups
NOTICE: ibd0: IBA promiscuous mode missed multicast group
NOTICE: ibd0: IBA promiscuous mode missed new multicast gid
SunOS 5.11 Last change: 20 Jul 2007 2
Devices ibd(7D)
Also, if the IBA SM indicates that multicast trap support is
suspended or unavailable, the system log contains a message
similar to:
NOTICE: ibd0: IBA multicast support degraded due to
unavailability of multicast traps
And when the SM indicates trap support is restored:
NOTICE: ibd0: IBA multicast support restored due to
availability of multicast traps
Additionally, if the IBA link transitions to an unavailable
state (that is, the IBA link state becomes "Down," "Initial-
ize" or "Armed") and then becomes active again, the driver
tries to rejoin previously joined groups if required.
Failure to rejoin multicast groups will trigger messages
like:
NOTICE: ibd0: Failure on port up to rejoin multicast gid
If the corresponding HCA port is in the unavailable state
defined above when initializing an ibd interface using
ifconfig(1M), a message is emitted by the driver:
NOTICE: ibd0: Port is not active
Further, as described above, if the broadcast-GID is not
found, or the associated MTU is higher than what the HCA
port can support, the following messages are printed to the
system log:
NOTICE: ibd0: IPoIB broadcast group absent
NOTICE: ibd0: IPoIB broadcast group MTU 4096 greater than port's
maximum MTU 2048
In all cases of these reported problems when running
ifconfig(1M), it should be checked that IBA cabling is
intact, an SM is running on the fabric, and the broadcast-
GID with appropriate properties has been created in the IBA
partition.
SunOS 5.11 Last change: 20 Jul 2007 3
Devices ibd(7D)
CONFIGURATION
The IPoIB service comes preconfigured on all HCA ports in
the system. To turn the service off, or back on after turn-
ing it off, refer to documentation in cfgadmib(1M).
FILES
/dev/ibd* special character device
/kernel/drv/ib.conf configuration file to start IPoIB
service
/kernel/drv/sparcv9/ibd 64-bit SPARC device driver
/kernel/drv/amd64/ibd 64-bit x86 device driver
/kernel/drv/ibd 32-bit x86 device driver
SEE ALSO
cfgadm(1M), cfgadmib(1M), ifconfig(1M), syslogd(1M),
gld(7D), ib(7D), kstat(7D), streamio(7I), dlpi(7P), attri-
butes(5), attach(9E)
NOTES
IBD is a GLD-based driver and provides the statistics
described by gld(7D). Note that valid received packets not
accepted by any stream (long) will increase when IBD
transmits broadcast IP packets. This happens because the
infiniband hardware copies and loops back the transmitted
broadcast packets to the source. These packets are discarded
by GLD and are recorded as 'unknowns'.
SunOS 5.11 Last change: 20 Jul 2007 4
|
