SENS/TORNADO 2.0 NETWORK STACK
CONFIGURATION/PERFORMANCE ISSUES
--------------------------------
1. INTRODUCTION
1.1 NETWORK STACK VERSIONS
1.2 NET POOLS
1.2.1 CLUSTERS
1.2.2 CLUSTER BLOCKS
1.2.3 MBLOCKS
1.3 RECEIVING: DATA FLOW FROM DRIVER TO APPLICATION LEVEL
1.4 SENDING: DATA FLOW FROM APPLICATION LEVEL TO DRIVER
1.5 SOCKET BUFFERS
1.5.1 UDP SOCKET BUFFERS
1.5.2 TCP SOCKET BUFFERS
2. NETWORK MEMORY RESOURCES SUMMARY
2.1 MEMORY POOLS
2.1.1 NETWORK STACK SYSTEM POOL
2.1.2 NETWORK STACK DATA POOL
2.1.3 NETWORK DRIVER POOLS
2.2 RECEIVE QUEUES
2.2.1 IP PROTOCOL RECEIVE QUEUE
2.2.2 IP FRAGMENT REASSEMBLY QUEUE
2.2.3 ARP RECEIVE QUEUE
2.2.4 TCP REASSEMBLY QUEUE
2.2.5 SOCKET RECEIVE QUEUES
2.3 SEND QUEUES
2.3.1 SOCKET SEND QUEUES
2.3.2 IP NETWORK INTERFACE SEND QUEUES
3. ESTIMATING NETWORK RESOURCES REQUIREMENTS
3.1 SOCKETS AND ROUTES
3.2 SENDING AND RECEIVING
3.2.1 LARGE SOCKET RECEIVE BUFFERS
3.2.2 TCP SMALL PACKETS
3.2.3 SENDING
3.2.4 RECEIVING
3.2.5 APPLICATION ERRORS
4. CONFIGURATION
4.1 NET POOL CONFIGURATION
4.1.1 NETWORK STACK MEMORY POOLS
4.1.2 NETWORK DRIVER MEMORY POOLS
4.2 PROTOCOL RECEIVE QUEUES
4.3 IP SEND QUEUES
5. REFERENCES
FIGURE 1. MEMORY POOLS AND QUEUES DEFAULT VALUES
1. INTRODUCTION
------------
This article provides background and guidelines for configuring
and performance-tuning the VxWorks network stack. This article
applies to the Tornado 1.0.1 stack with SENS installed, and to
the Tornado 2.0 stack with or without the Network Protocol
Toolkit product installed. The assumed network programming
interface is the BSD sockets interface; the zbuf sockets interface
is mentioned only in passing. Tornado/setup.log file shows
which version of Tornado and which components are installed.
The Tornado 2.0 VxWorks Network Programmer's Guide, Section
4.6.3, has the following comments regarding network stack memory
configuration:
"The values shown above are reasonable defaults, but the network
requirements for your system could be radically different. For
example, your use of the network stack could require more
clusters of a particular size. Making such a determination is a
matter of experimentation and analysis.",
"Change these constants after you fully understand what they do.
Setting inappropriate values can make the TCP/IP stack
inoperable.", and
"Carefully planned changes can significantly improve performance,
but reckless changes can significantly reduce performance."
As many users find out, the default values for the network stack
configuration parameters are adequate to download a file, open a
few sockets, add a few routes to the routing table, and run
simple client/server applications. Anything beyond that makes
the stack hang for a while and resume a short time later.
The network system requires clusters (blocks of memory) of
various sizes from different pools, used to store system
data structures, incoming data, and outgoing data. Clusters
carrying data are stored in various holding areas (queues)
along the path from application level to the network driver
and vice-versa. The number of clusters of each size and the
capacity of the holding areas must be configured to suit the
application needs. It is also necessary to ensure that the
clusters are processed regularly so that they will be available
to carry new data. If clusters are held in any area for too
long, the system's throughput will decrease. If the accumulation
continues, data flow will eventually stop until clusters are
freed up.
Analyzing application needs requires basic knowledge on how the
network system works. This paper will describe:
1. How data flows from application level to network interface driver
and vice-versa, for TCP and UDP;
2. The role of tNetTask;
3. How the configuration of network stack and driver memory pools,
application socket buffer sizes, protocol input queues, and
interface output queues interact with network performance.
Understanding these factors would help the reader set the network
stack configuration values appropriately. Tuning configuration
parameters for performance is in essence estimating the resources
needed to sustain data flowing at a certain rate. As in any other
inventory situation, requirements must be estimated taking into
account fluctuations in demand and delays. Having enough capacity
to handle the worst case is the only sure way to ensure reliability.
This approach may not be cost effective in many cases, but may be
the only alternative for some. Only users can make that
determination based on their application needs.
Section 1 presents background information on basic network stack
versions and terms, followed by a description of data flow for
receiving and sending, and socket buffers.
Section 2 presents a summary of the memory resources used by the
network system and a description of the various queues where these
resources wait is presented next.
Sections 3 and 4 present some guidelines for estimating requirements
and configuration information.
Section 5 lists resources where additional information can be found
in vxWorks Documentation, reference books and articles from:
WindSurf->Technical Solutions II.
1.1 NETWORK STACK VERSIONS
--------------------------
The network stack in Tornado 1.0.1 is based on BSD4.3, which is
essentially the network protocol IP over the link layer protocol
Ethernet. BSD4.3 Ethernet network interface drivers do not
support multicasting, or polled Ethernet for system mode debugging.
Under T101 the SENS component was optional. Under Tornado 2.0,
SENS is the one and only network stack. SENS features:
a. BSD4.4 network stack.
BSD4.4 has enhancements at the network stack level such as
support for CIDR (variable length subnet masks),
multicasting, and various protocols such as DHCP, and RIP.
b. MUX
The MUX interface facilitates connecting END drivers to
other user-ported network protocols besides IP. END drivers
can be written for link layer protocols besides Ethernet.
c. BSD4.4 and END drivers for supported Ethernet network cards.
BSD4.3 drivers were modified slightly to work with the BSD4.4
network stack. Either BSD4.4 or END drivers can be used for
booting. END drivers are required for multicasting and
system mode debugging over Ethernet. The SENS release
provided END Ethernet drivers for some BSPs. The Tornado 2.0
release provides END Ethernet drivers for all supported BSPs.
The free Network Protocol Toolkit (NPT) source product contains
updated documentation for porting to the END driver model
and source for the MUX and supporting network libraries.
In addition, NPT contains information for porting to the NPT
driver model which facilitates porting non-Ethernet drivers.
Network drivers developers should get the Network Protocol
ToolKit product from their sales representative. The
outdated BETA User's Guide is still available from
WindSurf->Documentation->Tornado 2.0 Documentation.
For sending, the path from IP to the physical medium can
take the BSD4.4 driver path, or the END driver path.
Similarly, for receiving, the BSD4.4 driver and the END driver
path both converge at the IP level. The details are hidden
in the initialization process.
The driver’s initialization process installs the proper function
pointers in the ifnet structure that IP keeps for every configured
interface. For example, ipAttach will install ipOutput as the
if_output function pointer of the ifnet structure for an END
driver. Similarly, ether_attach will install ether_output for
a BSD4.4 driver.
IP selects an interface to send a packet, based solely on the
routing table. IP calls the if_output function of the selected
interface, so IP does not need to know what kind of driver it is.
The setup.log file found in the Tornado base directory shows which
network stack version is installed:
SENS 1.0: T101 + WindNet SENS installed under SETUP 1.4
SENS 1.1: T101 + WindNet SENS installed under SETUP 1.5
Tornado 2.0: SETUP 2.0. The word SENS does not appear since
SENS is not an optional component anymore.
Note: T101 ARM BSPs have SENS 1.0 installed by default. There
was no SENS 1.1 patch update for ARM BSPs.
1.2 NET POOLS
---------
Tornado 1.0.1 (without the SENS component installed) had a memory
pool consisting of fixed data structures, BSD style mbufs, used
both to store small stack data structures such as socket addresses,
and for packet data. BSD mbufs were designed to facilitate passing
data between network drivers and the network stack, and contain
pointers that can be adjusted as protocol headers are added or
stripped. BSD mbufs contain space within them to store small
amounts of data. Larger amounts of data were stored in
fixed-sized clusters (typically 2048 bytes), which could be
referenced and shared by more than one mbuf.
In T101 + SENS and in Tornado 2.0, the BSD network buffer
implementation has been changed somewhat. Slightly modified BSD
mbufs are retained, but always reference external clusters rather
than carrying data directly; they are now called "mblocks." Pools
of clusters of multiple sizes are used, and a new structure called
the "cluster block" was added, supporting the zbuf sockets
interface and multiple network pools in addition to cluster
sharing.
The file target/h/netBufLib.h has the definitions of the new memory
system. The basic structure is the NET_POOL, which contains mblocks,
cluster blocks, and clusters of one or more sizes. The network stack
and some network drivers create different NET_POOLs:
a) The network stack DATA pool is used for actual data and headers
transferred between the application, the stack, and the network
interface.
b) The network stack SYSTEM pool is used to allocate clusters for
holding network stack control structures such as sockets,
protocol control blocks, routes, and so forth.
c) Network interface pools are used by (some) network drivers
to hold clusters for receiving packets off the wire, and for
staging packets to be transmitted by the target.
Users can create pools for their own purposes as well.
The header target/h/net/mbuf.h maps the Berkeley style mbuf
definitions to the new style.
1.2.1 CLUSTERS
--------
A NET_POOL may contain several cluster pools, each one holding
clusters of a single size. Valid cluster sizes range from
64 (2 to the 6th) to 65536 (2 to the 16th). However, not all
conceivable combinations of cluster pool sizes are allowed
within a net pool.
The NET_POOL structure contains an array of pointers to cluster
pools. Each row in the array corresponds to an allowed cluster
size range, measured in bytes, starting at a power of 2. The
array has room for a maximum of 11 entries, with indices ranging
from 0 (6 - minimum cluster size log 2 = 6 - 6) to 10 (16 - 6).
When configuring a NET_POOL, choose at most one cluster size
from within the range of each row, as given by this table:
Row 0: 64 <= cluster size <= 127
Row 1: 128 <= cluster size <= 255
Row 2: 256 <= cluster size <= 511
Row 3: 512 <= cluster size <= 1023
Row 4: 1024 <= cluster size <= 2047
...
Row 9: 32768 <= cluster size <= 65535
Row 10: 65536 = cluster size
If 2 cluster sizes fall in the same row, the pointer to the
second cluster will replace the first.
1.2.2 CLUSTER BLOCKS
--------------
A cluster block is a control structure pointing to a cluster.
It contains a count of the mblocks referencing the cluster,
as well as an optional function pointer and three function arguments.
If the function pointer (pClFreeRtn field) is non-null, the function
is called with the arguments when the cluster is freed; this is
used to support the zbuf sockets interface. Users can provide their
own functions as well. If the pClFreeRtn function pointer is null,
the network stack returns the cluster to the appropriate cluster
pool. The address of the cluster pool is found by subtracting 4
bytes from the clNode.pClBuf member of the cluster block.
One cluster block is required for each cluster.
1.2.3 MBLOCKS
-------
Mblocks are control structures which point to a section of data
within clusters, and link them together into packets and chains
of packets. Mblocks also contain other information about the
type of data contained in the clusters they reference. There
should be at least as many mblocks as there are cluster blocks.
Some extra mblocks are needed for the network stack DATA pool.
The stack uses DATA pool mblocks to reference data in the clusters
without copying. A TCP socket's send buffer is a chain of mblocks
linking together clusters containing data which has not yet been
acknowledged by the socket's peer endpoint. Some of this data
is also referenced by additional mblocks used to form the TCP
segments which are passed to the network driver for transmission.
In UDP, additional DATA mblocks are used to store source address
and port information for received UDP packets. A rough estimate
(sometimes erroneously treated as a hard rule) is that there should
be two times as many mblocks in the stack DATA pool as cluster
blocks. This "rule" certainly does not apply to the stack SYSTEM
pool or to network interface pools.
If the Network Protocol Toolkit product has not been installed,
the IP protocol driver allocates just two mbufs from the net
pool of each END interface it attaches to. Only the network
driver allocates from an END or NPT driver pool if NPT has been
installed.
BSD 4.4 drivers may allocate mbufs from the DATA pool.
When freed, mblocks, clusters, and cluster blocks are returned
to the pool from which they were allocated.
1.3 RECEIVING: DATA FLOW FROM DRIVER TO APPLICATION LEVEL
-----------------------------------------------------
Network drivers, and protocols like TCP and IP, schedule tNetTask
to do task level processing. IP and TCP schedule tNetTask to
process timers: to drop IP fragments that timed out before being
reassembled, and to send TCP delayed data acknowledgements, or
retransmissions. Drivers schedule tNetTask to process received
packets, and sometimes to perform miscellaneous other jobs.
While processing received packets, protocols may need to send
output messages (ICMP errors, TCP immediate ACKs or fast retransmits,
TCP data sent in response to peer's window openings, ARP replies,
and so forth). Together with timer-triggered TCP retransmissions
or resets, these are cases when sending is done in the context of
tNetTask.
tNetTask is not involved when applications call output functions
like sendto() or write(). Such processing is done within the
context of the application task. (However, TCP data placed into
a socket send buffer by an application's write() call may be later
transmitted in the context of tNetTask when the send window opens.)
Below is a overview of the receive processing done by tNetTask:
When a packet is received, within the driver's interrupt service
routine, netJobAdd() is called to add the driver's task level
receive routine to a ring buffer of jobs (fixed capacity = 85 jobs)
for tNetTask. This is done to minimize the time spent in the ISR
with interrupts locked. If the job is successfully added,
netTaskSemId is given to awaken tNetTask. If not, a ring buffer
overflow log message appears.
When tNetTask (priority 50) runs, it will first process all the
jobs in the ring buffer. If any IP datagrams have been queued,
it then calls ipintr() to process them, and finally calls arpintr()
if any ARP packets are queued. Each time tNetTask runs a job
from the ring buffer, and again before calling ipintr() and arpintr(),
it takes the splSemId semaphore by calling splnet() to lock out
other network stack tasks.
The driver's task-level receive function calls the stack receive
function. The driver lends the buffer to the stack, if possible.
The stack strips the packet's link layer header to determine who
should process the packet next. (NPT drivers do the header "parsing"
for the stack; for other drivers, the stack must do so itself.).
For END drivers, the MUX hands the packet to the appropriate protocol
input routine; for ARP and IP packets, this is ipReceiveRtn(). BSD
4.4 drivers call do_protocol (ether_input). ipReceiveRtn and
ether_input in turn call function do_protocol_with_type(), which
places the packet in the appropriate protocol queue
(IP = type 0x800), (ARP = 0x806), if there is room. The default
size for both the IP and ARP input queues is 50 packets. The
packet is dropped if the corresponding queue is full.
IP determines if the packet arrived in good condition and whether it
is a fragment or a full datagram. If it is a datagram, or if it
is a fragment completing a datagram, and the datagram specifies a
known transport-level protocol, IP then calls the appropriate protocol
receive function (e.g. tcp_input, udp_input, icmp_input ...).
The receiving protocol processes the packet. If there is room in
the socket receive buffer, the data is appended to the socket's
receive buffer. If a task was blocked waiting to read data from
the socket, it is given a semaphore; the task is then READY to
run.
When the priority level is such that the task runs and reads from
the socket, the socket layer copies the data to the buffer provided
by the user. When all the data from a cluster has been copied to
the application's buffers, the cluster is returned to the driver's
pool, or to the network stack DATA pool if a BSD4.4 driver had to
borrow from the network stack DATA pool. (If the zbuf interface is
used, the cluster is not returned to its pool until the received
zbuf is deleted.)
Some output may occur during processing of received packets.
IP may need to send various ICMP messages.
If the received packet is a UDP packet, and there is no socket bound
to the destination port, UDP will call icmp_error to send an
ICMP Unreachable error message.
For TCP, tcp_output() may be called during reception for a number
of reasons, for example, to send an immediate ACK if an
out-of-sequence segment is received, to do a fast retransmit, or to
send more available data, if the application is sending data, in
response to a window opening. (If the application is receiving data,
window updates may be sent to the peer after the application reads
data from the socket buffer, but this does not happen in the context
of tNetTask.)
The function ip_output(), and ultimately the driver's output function
are eventually called, still in the context of tNetTask.
1.4 SENDING: DATA FLOW FROM APPLICATION LEVEL TO DRIVER
---------------------------------------------------
In most cases transmission of application data to the network
occurs in the context of application tasks. tNetTask is involved
in just a few cases: tNetTask may transmit TCP application data,
both during retransmission, and when data earlier placed in the
socket buffer by the application is later sent in response to
the send window opening; and tNetTask may transmit any IP datagram
held earlier pending an ARP reply.
When send(), sendto(), or write() is called at the application
level, the following happens:
splSemId semaphore is taken by calling splnet to lock out other
network stack tasks.
If there is room in the socket send buffer, the socket layer
copies data from application level buffers to clusters allocated
from the network stack DATA memory pool (_pNetDpool).
For example, if an application writes 1460 bytes, the socket layer
will request a buffer big enough to contain the data plus some
space for link layer headers. The system will return the closest
size available. The data will be copied to a single buffer (2048
bytes, in the default configuration), if available, or to a chain
of smaller buffers, otherwise.
If there are no buffers, the system attempts to find buffers,
and if it is not successful sendto, write ... will return
ENOBUFS for non-blocking AND blocking sockets.
The socket layer then calls the protocol user request function
for output, which in turn calls the actual protocol output function
(e.g., udp_output(), tcp_output()) to process the data.
After the appropriate protocol processes the data and adds its
header, one or more packets may be passed to IP.
Based on the destination IP address, IP locates the appropriate
route and calls the output function for the outgoing interface
corresponding to that route. If the packet is too large for
the outgoing interface's MTU, IP will fragment it before calling
the interface output function.
The output function will resolve the address to obtain a hardware
address, if necessary. It will then add the packet to the send
queue for the appropriate interface, if there is room. (When a
network interface is attached to IP, the default size for the
send queue is set at 50 packets.) ENOBUFS is returned if there
is no room in the send queue, and the packet is dropped.
The appropriate if_start() function is called to pass every packet
in the interface send queue to the driver's transmit function.
After all packets have been passed to the driver (or the driver
has stopped accepting packets until a transmit restart) splx() is
called to release splSemId.
Each driver (END or BSD) is different. Some END drivers have
clusters for sending, others use different data structures called
various names such as descriptors or frames. Some BSD drivers
have 1 transmit frame, others have many.
Whether the driver uses clusters or frames, it still has to find
a free one.
In general for END DRIVERS, using clusters:
Within the driver's output function, the driver takes its own
transmit semaphore which is SEM_INVERSION_SAFE
The driver attempts to allocate a buffer from its pool to copy
the packet passed by the stack. (The copy is done for two reasons:
the hardware may not support gather-write, and the stack buffers
containing the data may not be cache-coherent.) If the driver does
not have enough buffers, it returns END_ERR_BLOCK. The stack will
prepend the packet to the send queue and try again the next time
a packet is queued for the interface, or after the driver calls the
MUX transmit restart routine. If the driver has enough buffers, the
driver frees the network stack DATA buffer. It then transmits the
packet. The driver gives its transmission semaphore.
UDP frees the network stack DATA buffers as soon as the message is
sent.
TCP, on the other hand, holds the DATA buffers for possible
retransmission until it receives an acknowledgement from the peer
TCP.
1.5 SOCKET BUFFERS
--------------
Socket buffers are used to store messages and other information
required by the protocols. As compiled, the maximum size for a
socket receive or send buffer is 258,111 bytes.
When a new UDP socket is created, its receive and send buffer are
set to the value of the global variables udp_recvspace and
udp_sendspace respectively. Similarly, TCP sockets buffers are
set to the value of the global variables tcp_recvspace and
tcp_sendspace.
The values of the global variables are:
-> tcp_sendspace
tcp_sendspace = 0xf9e00: value = 8192 = 0x2000
-> tcp_recvspace
tcp_recvspace = 0xf9e04: value = 8192 = 0x2000
-> udp_sendspace
udp_sendspace = 0xfec74: value = 9216 = 0x2400
-> udp_recvspace
udp_recvspace = 0xfec78: value = 41600 = 0xa280
Another value set when a socket is created, or changed when
setsockopt is called with options SO_SNDBUF/SO_RCVBUF, is the
maximum number of mbufs (in bytes) that a socket may hold. mbufs
is the same as clusters. The limit is set to avoid having a
single socket consume too many mbufs and is calculated as follows:
max number of mbufs = minimum ( 5* maximum buffer size, 256*1024)
(in bytes)
For both UDP and TCP, space in the send or receive buffer is
calculated using the formula below prior to sending or receiving:
space = min (buffer maximum size - total bytes in buffer),
(max number of mbufs - total mbuf bytes in
buffer)
To change the default buffer sizes for all sockets, change the
value of the global variables from the shell or programmatically.
Use setsockopt to change the value for an individual socket.
Calling setsockopt to set the maximum size for buffer space just
declares what the limit will be. No memory is actually allocated
until a packet is sent or received.
1.5.1 UDP SOCKET BUFFERS
------------------
For sending, if the message size is larger than the socket send
buffer, EMSGSIZE error is returned.
For receiving, since UDP is connectionless, UDP stores the
message source IP address and port in the receive buffer.
Therefore, an additional 16 bytes per message are needed for the
receive buffer. If there is no space, as determined by the space
calculation above, received packet is dropped and the error is
reported by udpstatShow as full sockets error.
For example, for a 172 byte UDP message received over Ethernet:
The 172 byte message is within a buffer of size 1520. The
address and port information is stored in a separate 128 byte
buffer. Each buffer requires a control data structure, an mblock
(size = 32 bytes):
# mbuf bytes = packet mbuf bytes + address information mbuf bytes
# mbuf bytes = 1520 + 32 + 128 + 32 = 1712 bytes
For a 2000 byte UDP message, since the maximum UDP message size
over Ethernet is 1472 bytes (1500 - 28 byte header), 2 packets are
received:
# mbuf bytes = (1520 + 32) * 2 + 128 + 32 = 3264 bytes
1.5.2 TCP SOCKET BUFFERS5
------------------
For TCP sockets, the socket receive/send buffer sizes determine
the maximum send and receive window sizes for the connection.
setsockopt must be called before the listen call on the server,
and before the connect call on the client, since maximum window
sizes are among the parameters exchanged when the connection is
first established. Increasing the buffers later has no effect.
Periodically TCP sends window updates informing the peer how much
data it can receive. The advertised window size is the available
room in the receive buffer. The window size cannot exceed the
maximum set when the connection was established, but can decrease
to 0 if the socket is not read, and the receive buffer becomes
full.
To improve TCP performance, Richard Stevens (UNIX Network
Programming Volume 1, Second edition, Page 192), suggests that:
1. socket buffer sizes should be at least 3 times the MSS.
2. buffer sizes should be an even multiple of the MSS for the
connection and
3. Buffer sizes should be at least as large as the product of the
bandwidth (bytes/sec) and the round trip time to keep the
"pipe" full and get the expected performance. Round trip times
(RTT) are obtained by using ping. For example, if the bandwidth
is 1000 bytes per second, and the average round trip time is 2
seconds, buffer sizes would have to be at least 2000 bytes.
For Ethernet, and an average RTT = 50 msecs:
10000000/8 bytes * 0.050 seconds = 62500 bytes which is
approximately 42.8 * MSS. Following 1 and 2 above, socket
buffer size should be 64240 (44*1460).
On an Ethernet which is not overloaded, RTT is much faster than
50 msecs.
The window size field in the TCP header is an unsigned short
(maximum = 65535 bytes). The Tornado 2.0 stack implements RFC1323,
the window scale option (See netLib.h), which is designed to get
around the 65535 limit for high speed connections (ATM) or slow
media (satellite) which require large window sizes.
2. NETWORK MEMORY RESOURCES SUMMARY
--------------------------------
There are 3 memory pools consisting of clusters and their control
structures, which are allocated at initialization time. These
pools cannot be increased later:
NETWORK STACK SYSTEM POOL
NETWORK STACK DATA POOL
NETWORK DRIVER POOLS
There are 7 types of queues (linked lists) that hold data waiting
to be processed. The data is contained in clusters previously
allocated from the DATA or the DRIVER pools. Adding a cluster to
a queue is just chaining the cluster to the queue's linked list.
The stack sets default values for the maximum length of some
queues, which can be changed after initialization:
RECEIVE QUEUES:
1. IP PROTOCOL RECEIVE QUEUE
2. IP FRAGMENT REASSEMBLY QUEUE
3. ARP RECEIVE QUEUE
4. TCP REASSEMBLY QUEUE
5. SOCKET RECEIVE QUEUES
SEND QUEUES:
1. SOCKET SEND QUEUES
2. IP NETWORK INTERFACE SEND QUEUES
2.1 MEMORY POOLS
------------
1. NETWORK STACK SYSTEM POOL
-------------------------
This pool is used to store data structures such as sockets
and routes. The default values are:
------ CLUSTERS REQUIRED -----
CLUSTER SIZE TOTAL PER TCP PER UDP PER ROUTE
BYTES SOCKET SOCKET ARP MAX **
NUM_SYS_64 40 0 0 2 3
NUM_SYS_128 40 1 1 0 0
NUM_SYS_256 40 1 0 1 2
NUM_SYS_512 20 1 1 0 0
**: Route requirements vary. For example, the first time
a network route is added with a new gateway,
(Example; routeAdd("147.11.48.0", "147.11.41.254")
3 64 and 2 256 clusters (3,0,2,0) are allocated. If
another route is added using the same gateway,
(Example: routeAdd("147.20.30.40", "147.11.41.254")
(1,0,1,0) clusters are allocated. If an ARP entry is added
with arpAdd, (2,0,1,0) clusters are allocated. routeAdd,
arpAdd themselves open a raw socket (0,1,0,1) temporarily
to pass information to the stack.
2. NETWORK STACK DATA POOL
-----------------------
This pool is used to copy data passed by applications
(sendto, write...) from application level buffers to network
stack clusters. The clusters are passed to the
protocols and ultimately to the network drivers. Protocols
(ICMP, TCP, RIP, ...) send messages too.
The stack allocates a cluster big enough to hold the message
plus some extra space for protocol headers. Smaller
clusters would be allocated and chained if the right size is
not available.
CLUSTER SIZE TOTAL
BYTES
NUM_64 100
NUM_128 100
NUM_256 40
NUM_512 40
NUM_1024 25
NUM_2048 25
For example, if an application calls "write" to send one
1460 byte message for a TCP socket, 1 2048 byte cluster
would be allocated from the pool. TCP would hold this
cluster in the socket send queue until the recipient
notifies that the data was received. Clusters used by UDP
are freed as soon as the driver copies the data to its own
clusters.
3. NETWORK DRIVER POOLS
--------------------
Tornado 2.0 offers 2 driver versions for each supported
network card: BSD 4.4 and END. Either type can be used,
but END drivers are required for multicasting and for
system mode debugging over Ethernet.
END drivers are configured by default in Tornado 2.0.
muxShow displays configured END drivers. ifShow displays
both END and BSD4.4 drivers.
Drivers set up their own pools of non-cacheable memory and
allocate buffers from it which are used for sending and
receiving. Buffers can be clusters or other data structure
types called frames or descriptors.
Drivers copy data passed by the stack to their own buffers
for transmission. Devices receive packets in the driver's
buffers. END and BSD4.4 drivers (some, not all) lend
buffers containing received packets to the network
stack to avoid copying.
Buffers have to be big enough to receive or transmit the
maximum packet size allowed by the link layer. For
Ethernet 10 Base T, the maximum size is 1518 bytes. Two
extra bytes are required to align the IP header on a 4 byte
boundary for incoming data. The default size of end driver
cluster pools vary, but the number could be as few as 40
1520 clusters total.
In this article, when it comes to drivers, clusters are
a generic name for buffers: descriptors, frames, or actual
cluster data structures.
Drivers differ:
END DRIVERS
-----------
Some END drivers have frames for transmission and clusters
for reception. Some END drivers have clusters for both.
END network drivers lend all their clusters.
BSD 4.4 DRIVERS
---------------
BSD drivers have frames. BSD drivers borrow mblocks and
cluster blocks from the network stack DATA pool when they
lend their frames to the stack. BSD drivers may borrow
clusters as well if they have lent the configured maximum
number of LOAN frames to the stack. Some BSD drivers
always borrow clusters.
2.2 RECEIVE QUEUES
--------------
1. IP PROTOCOL RECEIVE QUEUE
-------------------------
This queue stores DRIVER or DATA cluster containing incoming
packets passed to IP from all configured interfaces.
IP processes the packets and passes them to the protocols
(TCP, UDP...). Protocols process the packets and queue
data in the destination socket receive buffers. If packets
come in bursts and the queue is full, the excess is
dropped.
The default value for this queue is 50 packets.
2. IP FRAGMENT REASSEMBLY QUEUE
----------------------------
This queue stores DATA or DRIVER clusters that contain
fragments of a larger IP datagram waiting to be reassembled.
The maximum size for an IP packet is 65535 bytes. Packets
exceeding the maximum transmission unit (MTU) for a link
layer protocol must be fragmented.
For example, for Ethernet 10BaseT, the maximum packet size
is 1518 bytes. Subtracting the Ethernet header and checksum
(18 bytes), the IP header (20 bytes), and the UDP header
(8 bytes) leaves 1472 bytes. A UDP message consisting of
1473 bytes arrives in 2 separate IP packets.
There is a default timeout to drop the fragments if they
cannot be reassembled in time. There is no explicit
queue size. IP fragments are dropped also when a pool
is out of clusters and the system drains the protocols.
3. ARP RECEIVE QUEUE
-----------------
This queue holds DATA or DRIVER buffers containing ARP
packets which are used for hardware address resolution.
If packets come in bursts and the queue is full, the
excess is dropped.
The default value for this queue is 50 packets.
4. TCP REASSEMBLY QUEUE
--------------------
This queue contains DATA or DRIVER clusters containing TCP
segments that arrive out of order.
For example, the maximum TCP message size over Ethernet
10BaseT is 1460 bytes (assuming no TCP options).
TCP avoids IP fragmentation, so each segment is 1460
bytes maximum.
The socket receive/send buffer sizes determine the
maximum send and receive window sizes for the connection.
If socket receive buffers are large, the peer TCP will be
able to send a large number of segments within the window.
TCP has to pass data to the application in the order in
which it was sent. When packets arrive out of order, or
are lost, TCP stores these segments in a reassembly queue
until the missing segment arrives.
There is no limit on the queue size nor is there a timeout
period to drop the fragments.
5. SOCKET RECEIVE QUEUES
---------------------
Each socket's receive buffer stores the DATA or DRIVER
clusters until the application reads the socket, at which
point, the clusters would be freed. This is important:
Driver or DATA clusters are freed IF AND WHEN applications
read the sockets. If they don't, data flow may stop
eventually.
The default value is 8192 bytes for TCP sockets and 41600
bytes for UDP sockets.
2.3 SEND QUEUES
-----------
1. SOCKET SEND QUEUES
------------------
Each TCP socket's send buffer holds DATA clusters
containing packets waiting to be acknowledged. The default
value for TCP sockets send queues is 8192 bytes.
UDP sockets send queues do not hold any data, but the stack
checks that the message size does not exceed the send socket
buffer size (9216 bytes). A DATA cluster is freed as soon
as the driver copies the data to its own clusters prior to
transmission.
2. IP NETWORK INTERFACE SEND QUEUES
--------------------------------
IP has a send queue for each configured interface which
stores the outgoing packets, DATA buffers, that IP needs to
pass to the driver for transmission. If a driver is busy
transmitting, and this queue is full, the packet is dropped
and ENOBUFS error is returned to the application. The default
value is 50 packets.
FIGURE 1: MEMORY POOLS AND QUEUES DEFAULT VALUES
--------------------------------------
RAW SOCK SEND BUFF SYSTEM POOL
****************** ******************
* DATA *-----------> * SOCKETS *
* CLUSTERS * | * ROUTES *
* 8192 bytes * | * PCBs, TCPCB *
****************** | ******************
IP SEND QUEUE
UDP SOCK SEND BUFF FOR "FEI" END DRIVER
****************** ******************
* DATA CLUSTERS *--->* DATA CLUSTERS *---------->
* 9216 bytes * * 50 PACKETS * |
****************** ****************** |
|
IP SEND QUEUE |
TCP SOCK SEND BUFF FOR "ELT" BSD 4.4 DRIVER |
****************** ****************** |
* DATA CLUSTERS *--->* DATA CLUSTERS *----------|--------
* 8192 bytes * * 50 PACKETS * | |
****************** ****************** | |
| |
******************** | |
* ARP RECEIVE QUEUE*<-<--------|-------|<
**** 50 PACKETS *** | | ||
| V ||
UDP SOCK RCV BUFF IP FRAGMENT QUEUE | FEI END DRIVER ||
****************** *************** | ************** ||
* DRIVER, DATA * * DRIVER, DATA* |<*END DRIVER * ||
* 41600 bytes *<---*CLUSTERS/FRAM* * * ||
****************** *************** *RECV: * ||
| ^ --*CLUSTERS: 64* ||
<----------------- | | *CLBLKs: 64 * ||
| | | *MBLK: 128 * ||
RAW SOCK RECV BUFF IP RECEIVE QUEUE | *TX FRAMES:32* ||
****************** *************** | ************** ||
* DRIVER, DATA * *DRIVER,DATA *<--| ||
*CLUSTERS/FRAMES *<---*CLUSTERS/FRAM* ||
* 8192 bytes * * 50 PACKETS *<-- ELT BSD4.4 DRIVER||
****************** *************** | ************** ||
| | | *BSD44 DRIVER* ||
<--------------- | | | * * ||
V V | *FRAME * ||
TCP SOCK RECV BUFF TCP REASSEMBLY Q. |-*DESCRIPTORS:*<- |
****************** *************** * Rx: 20 * |
* DRIVER, DATA *<---* DRIVER,DATA * * Tx: 1 *--->
* 8192 bytes * *CLUSTERS/FRAM* * Loan: 16 *
****************** *************** **************
3. ESTIMATING NETWORK RESOURCES REQUIREMENTS
Network stack memory exhaustion problem is not unique to vxWorks.
TCP and UDP do not prioritize sockets. There are no facilities
to identify sockets which could be considered less important
and which could be targeted for selective drops, or even
closing, when resources are limited.
Packets will be dropped during peak times if there are not enough
driver buffers to lend to the stack to support the sum of the UDP
and TCP socket receive buffers for the expected number of
concurrent socket connections. This sum could be a very large
number.
Even if there is a single socket, there could be still be high
demand. For example, applications expecting to receive bursts
of 100 UDP packets even if they carry only one byte of data
will still be consuming 100 driver buffers.
Similarly, applications will hang and resume after a while if
there are not enough SYSTEM clusters to open sockets or create
routes for applications that generate a high demand for them.
3.1 SOCKETS AND ROUTES
------------------
If the stack does not have enough SYSTEM clusters, applications
like FTP and Web servers can hang for a while when there is not
enough memory to open sockets. iosFdShow displays open sockets.
inetstatShow displays socket status. When TCP on the target
closes a socket, the socket is removed from the file descriptor
table, but the memory used by that socket is not de-allocated
until TIME_WAIT time elapses (about 1 minute). The TCP protocol
needs this time to minimize the chance that duplicate packets
belonging to an earlier connection will be delivered to a new TCP
connection. inetstatShow displays the socket states.
If a routing protocol like RIP is configured, or if, for example,
the target is not given an explicit subnet mask, when necessary,
the route table can get very large since the target is really
listening to ARP broadcasts for all subnets. A message may
appear on the on the console: arpresolve: can't allocate llinfo
netStackSysPool can be used to tune the SYSTEM pool. See
5.1.1 NETWORK STACK MEMORY POOLS for details.
3.2 SENDING AND RECEIVING
---------------------
Some conditions that would generate a high demand for driver
buffers:
3.2.1 LARGE SOCKET RECEIVE BUFFERS
----------------------------
A UDP or TCP message consisting of a single byte of user data
will be received in a whole driver buffer. Large socket receive
buffers can contain many driver buffers.
UDP messages larger than the MTU (For Ethernet = 1472
bytes) are fragmented by IP. IP on the receiving side will have
to reassemble the fragments. Each fragment will be received on a
separate driver or DATA buffer.
TCP does not send packets larger than the MTU. However, if
socket receive buffers are large, the peer TCP will be able to
send a large number of segments within the window. Since TCP has
to pass data to the application in the order in which it was
sent, when packets arrive out of order, or are lost, TCP stores
these segments in a reassembly queue until the missing segment
arrives.
TCP has no provision to drop packets in the reassembly
queue, so driver buffers can sit there for a long time or
indefinitely. TCP on the receiver side sends ACKS for the
missing segment every time it gets an out of sequence packet.
TCP on the sender side retransmits the missing segment, or
even starts over again retransmitting the first unacknowledged
segment if it does not get any response. TCP on the sender
side retries a number of times but eventually it gives up.
This situation could happen if all of the driver buffers are
held up in the TCP reassembly queue, and there are no available
driver buffers for transmission or to receive the missing
segment.
3.2.2 SMALL TCP PACKETS
-----------------
TCP normally does not send small packets, but many users enable
TCP_NODELAY to force TCP to do so.
Enabling TCP_NODELAY on the sender side (See entry for setsockopt in
vxWorks Reference Manual) prevents TCP from waiting to collect
small packets until an MSS sized packet can be sent.
For TCP over Ethernet: MSS = 1460 bytes. TCP on the sender side
still is subject to the constraint that it must not send more
than the peer's advertised window.
It should be noted that enabling TCP_NODELAY on the receiver side
will not have a direct effect on how fast the receiver side
acknowledges data. When an application reads a socket, the TCP
on the receiver's side calculates whether it should tell the peer
that is OK to send more. It does so by sending ACKS. This ACK
is not immediate. The following calculation takes place:
Compare available window to amount of window known to peer. If
the difference is at least two maximum size segments
(For TCP = 1460 bytes) or at least 50% of the maximum possible
window, then send a window update to peer. Otherwise, the ACK
gets delayed until the FAST TIMER goes off about 200 msecs later.
TCP works this way following RFCs to avoid contributing to
network congestion.
If the application message size is small (less than 1460 bytes),
reducing the socket receive buffer size would make ACKS come
faster.
If the application message size is larger than MSS
(1460 bytes), you should test various message sizes
(1 MSS, 2 MSS, ...) to find a level that gives good
performance. Surprisingly, large buffers are not always
better. TCP ACK-every-other-segment feature improves
performance since the 200 msec delay would not occur as often.
It should be noted that TCP was developed to maximize
reliability, not performance. TCP wants to be efficient
also, so it tries not to add unnecessary packets on the
network thereby increasing congestion. The THEORETICAL
maximum throughput, under ideal conditions, for a single
TCP socket connection on 10 BASET, is 1,183,667
bytes/second. In reality, there are delays. Achieving
90% of this maximum is considered very good.
TCP can be tuned to give a reasonable performance, but
the reader should understand its limitations by consulting
TCP/IP reference books.
3.2.3 SENDING
-------
Sending a lot of packets requires the following steps:
STEP 1: INCREASE THE NETWORK STACK DATA POOL
------------------------------------
netStackDataPoolShow displays the status of
the DATA pool. If there are no free buffers
or if the statistics for the number of times
failed to find space is greater than 0, increase
the appropriate cluster pools based on the
application message size. Detailed instructions
can be found in Section 4.1.1 NETWORK STACK
MEMORY POOLS.
STEP 2: INCREASE THE NETWORK DRIVER BUFFER POOL
---------------------------------------
See Section 4.1.2 NETWORK DRIVER MEMORY POOLS.
STEP 3: INCREASE THE NETWORK INTERFACE SEND QUEUE
-----------------------------------------
See Section 4.3 IP SEND QUEUES
3.2.4 RECEIVING
---------
Receiving lots of packets for many socket connections requires
the following steps:
EITHER
STEP 1: INCREASE THE NETWORK DRIVER BUFFER POOL
---------------------------------------
If an application will be using the network
intensively, performance will be enhanced if the
network interface driver is configured with a
sufficient number of buffers so that packets
are not dropped unnecessarily. See Section
4.1.2 NETWORK DRIVER MEMORY POOLS.
A utility like endPoolShow, found in attached
utilities.c can help the user find the optimal
number for END drivers. For BSD4.4 drivers, there
is no specific function that can show how many
times the driver was out of its own buffers and
had to borrow from the network stack DATA pool.
However, netStackDataPoolShow can be helpful.
Both endPoolShow and netStackDataPoolShow would
display if buffers were exhausted at any time.
STEP 2: INCREASE IP INPUT QUEUE
-----------------------
See Section 4.2 PROTOCOL RECEIVE QUEUES.
OR
STEP 1: DECREASE SOCKET RECEIVE BUFFERS
STEP 2: DECREASE THE NUMBER OF CONCURRENT SOCKET CONNECTIONS
STEP 3: ADJUST APPLICATION MESSAGE SIZE AS NECESSARY TO BE
CLOSER TO THE MTU TO IMPROVE EFFICIENCY.
3.2.5 APPLICATION ERRORS
------------------
The network driver buffer mechanism can be disrupted if tNetTask
cannot process packets, or applications errors have deadlock
conditions which prevent the sockets from being read.
The priority of applications that need the network should be lower
(>50) than tNetTask's (= 50). User's applications usually run at
priority 100. Call i from the shell to display priorities for
running tasks.
tNetTask takes splSemId which is SEM_INVERSION_SAFE. Do not create
and then take SEM_INVERSION_SAFE semaphores before making a socket
call or your task could be promoted to run at tNetTask level. See
priority inversion discussion in the vxWorks Programmer's Guide
(Section 2.4.3). If priority inversion occurred, the command i
would show it as 0+I:
->i
tNetTask netTask 807c9750 0+I PEND
Application may have deadlock conditions which prevent them from
reading sockets. If inetstatShow (or equivalent in other
systems) displays data backed up on the send side
and on the receive side of the peer, most likely there is a
deadlock situation within the client/server application code.
TCP is a stream of bytes. Loops are required to read or write,
since TCP can return partial reads and partial writes.
Running both server and client in the target by sending to
127.0.0.l or to the target's own IP address is a good way to
detect this kind of problem.
4. CONFIGURATION
See WindSurf->Technical Articles II:
How to build images with network support in Tornado 2.0
which contains instructions for building a kernel with network
show libraries and a target shell whose output goes to serial
console. The target shell can be used to debug network problems
when network problems occur.
The attached utilities.c file contain code for some utilities which
are not part of the API:
endPoolShow
protocolQValuesShow
changeProtocolInputQMax
ifQValuesShow
changeIfSendQMax
4.1 POOL SHOW FUNCTIONS
-------------------
Call pool show functions to find the level where resources are
adequate:
netStackSysPoolShow
netStackDataPoolShow
endPoolShow (found in utilities file).
Pool show functions display:
number of times failed to find space: If not 0, pool is too
small.
number of times waited for space: 0
number of times drained protocols for space: IP fragments are
dropped when
drivers or the
stack are out
of clusters.
Usage: the cumulative number of times clusters of each size
have been allocated.
Clusters: Total clusters: Free + clusters in queues.
Mbufs and their use.
Increase clusters, cluster blocks and mblocks as necessary until
there are no shortages.
4.1.1 NETWORK STACK MEMORY POOLS
--------------------------
Increase the default values as follows:
1. INCREASE NUM_FILES
The socket library uses NUM_FILES to configure the size of
the socket file descriptor table. The default value for
NUM_FILES is 50 sockets.
2. TCP SOCKETS
For every TCP socket, the system allocates 3 clusters
(1 128 byte, 1 256 byte, and 1 512 byte clusters) used for
the generic protocol control block (inpcb), TCP protocol
control block, and the socket structure respectively.
3. UDP SOCKETS
For every UDP socket, the system allocates 2 clusters
(1 128 byte, 1 512 byte) used for the inpcb and socket
structures respectively.
4. ROUTES
For every link layer route (ARP), 3 clusters (2 64 byte,
and 1 256 byte) clusters are allocated.
BOOTROM IMAGE MACROS
--------------------
The macros with MIN suffix (i.e. NUM_SYS_64_MIN... and
NUM_64_MIN...) are used for BOOTROM images which usually need
to be small. The values for the MIN macros are usually
adequate to download the runtime image and do not normally
need to be changed.
RUNTIME IMAGE MACROS
--------------------
The network stack SYSTEM POOL macros are NUM_SYS_64,
NUM_SYS_128, ...etc.
The network stack DATA pool MACROS are NUM_64, NUM_128...etc.
TORNADO 101 BUILD METHOD
------------------------
a. Increase NUM_FILES in target/config/all/configAll.h.
b. Increase the default network stack memory sizes in
netBufLib.h.
PROJECT FACILITY BUILD METHOD
------------------------------
a. Increase NUM_FILES
Right Click on
network components->
networking protocols->
BSD SOCKET to bring up the properties window. Click on
Params window to change NUM_FILES.
b. Change values for the runtime image in the Params
window for network buffer initialization component:
network components->
basic network initialization components->
network buffer initialization->
4.1.2 NETWORK DRIVER MEMORY POOLS
---------------------------
The size of a driver's pool is determined by the parameters
passed to the driver, during initialization time
(See the VxWorks Reference entry for the driver, since each one
is different.
Drivers have a ring of buffers for transmission and reception.
For some drivers, the ring size may be limited by the hardware to
a maximum.
BSD 4.4 DRIVERS
---------------
The LOAN parameter allows having extra buffers for loaning so the
ring always has enough buffers.
END DRIVERS
-----------
END drivers lend all their clusters. If END driver's
cluster pools are depleted, reception will stop until
the clusters are freed by the network stack or the applications
IF and WHEN socket buffers are read. Transmission may stop as
well if the driver uses clusters for transmission.
For end drivers, target/config/bsp/configNet.h is the place to
change the default parameters in the driver's load string. For
BSD4.4 drivers, the parameters are passed to the attach function.
See basicNetConfiguration document for more details.
The number of buffers is typically the number of transmit
descriptors plus the number of receive descriptors plus the number
of loan buffers. If the loan parameter is not part of the
initialization string, then changing the LOAN parameter in the
driver's header file will have no effect unless you have source
for the driver and can recompile the object. Some drivers just
make loan buffers equal to the number of receive or send
descriptors.
Example: PPC target uses dc END driver.
In configNet.h
/* <devAdrs>:<PCIadrs>:<ivec>:<ilevel>:<numRds>:<numTds>:<memBase>:<memSize>:<userFlags> */
#ifdef ULTRA
# define DEC_LOAD_STRING "0x81020000:0x80000000:0x09:0x09:-1:-1:-1:0:0x0"
#else
# define DEC_LOAD_STRING "0x81020000:0x80000000:0x0a:0x0a:-1:-1:-1:0:0x0"
#endif
>From dec21x40End manual entry:
The default -1 means use 32 buffers
So I changed the default numbers to 128 each:
#define DEC_LOAD_STRING "0x81020000:0x80000000:0x0a:0x0a:128:128:-1:0:0x0"
NUM_LOAN is declared in dec21x40End.h as 16.
The cluster pool available to the driver will be:
numRds + numTds + NUM_LOAN = 128 + 128 + 16 = 272
The receive and transmit numbers are specified separately, but
the clusters in the pool are in fact used for both sending and
receiving, and all buffers are loaned.
-> endPoolShow("dc",0)
type number
--------- ------
FREE : 510
DATA : 2
HEADER : 0
SOCKET : 0
PCB : 0
RTABLE : 0
HTABLE : 0
ATABLE : 0
SONAME : 0
ZOMBIE : 0
SOOPTS : 0
FTABLE : 0
RIGHTS : 0
IFADDR : 0
CONTROL : 0
OOBDATA : 0
IPMOPTS : 0
IPMADDR : 0
IFMADDR : 0
MRTABLE : 0
TOTAL : 512
number of mbufs: 512
number of times failed to find space: 0
number of times waited for space: 0
number of times drained protocols for space: 0
__________________
CLUSTER POOL TABLE
_________________________________________________________
size clusters free usage
1520 272 144 450
---------------------------------------------------------
4.2 PROTOCOL RECEIVE QUEUES
-----------------------
There are 2 global variables that control the queue length for
the ARP queue and the IP queue: arpintrq, and ipintrq. The queues
hold the unprocessed ARP and IP packets. Packets are dropped if
the queue length is at the maximum level. The default value for
each is 50. No error statistics are increased.
If building the Tornado 101 way, change the default value for
ipintrq.ifq_maxlen by changing the value for IP_QLEN_DFLT in
target/h/netLib.h. If building using the Tornado 2.0 GUI:
network components
networking protocols
core TCP/IP components
Right Click on IPv4-> Properties-> Parameters
The default values and drop statistics are displayed by
protocolQValuesShow. Change the default value for ARP or IP at
run time calling changeProtocolInputQMax. Neither is an API
function. See utilities.c.
4.3 IP SEND QUEUES
--------------
When an END, BSD4.4 driver, or the loopback driver is attached
to IP, one of the parameters set by IP is the size of the maximum
send queue length.
On sending, ipOutput for END drivers or ether_output for BSD4.4
drivers attempts to add a packet to interface's send queue, one of
the fields of the ifnet structure that each driver creates
(See target/h/net/ if.h). Default queue size is 50 packets.
Packet is dropped if queue is full, the ifq_drops statistics
increases and the socket call returns ENOBUFS.
Call ifQValuesShow to display the current values and drop
statistics.
Use changeIfSendQMax to increase the default maximum, if an
application will be sending more than 50 packets in rapid
succession. ifQValuesShow and changeIfSendQMax utilities can be
found in the code section.
5. REFERENCES
----------
5.1 VXWORKS DOCUMENTATION
---------------------
The Tornado 2.0 Documentation Collection has the most up-to-date
manual versions. Manuals are available from the Help Menu in
Tornado and also from WindSurf.
In addition to manuals, WindSurf has Patches, Topic keyword
search engine, and Networking Demo Code among other things.
www.windriver.com -> Support -> WindSurf
Access to WindSurf requires a password. Register at:
http://www.windriver.com/corporate/support/private/validate.html
Network documentation is distributed in the following manuals:
TORNADO USER'S GUIDE
Chapter 1: Overview. Description of Tornado IDE.
Chapter 2: Setup and Startup. Host and target setup.
Chapter 4: Projects. Project Facility Build Method.
Chapter 5: Target Server. Tools/target interface.
Chapter 6: Target Shell. Host-based shell: WindSh.
Appendix F: FTP Server. Host system FTP server setup.
TORNADO RELEASE NOTES
Section 7: Host Development Environment Enhancements
Section 9.1: Enhanced Network Stack
Section 11: Backwards Compatibility Issues
VXWORKS PROGRAMMER'S GUIDE
Chapter 8: Configuration and Build. Tornado 101 Build Method.
Chapter 9: Target Shell.
VXWORKS NETWORK PROGRAMMER'S GUIDE
Chapter 13: Booting over the Network.
Chapter 14: Upgrading 4.3 BSD Network Drivers.
Network drivers developers should get the Network Protocol
ToolKit product from their sales representative. NPT is the most
up to date manual on this subject. The BETA User's Guide is
available from WindSurf which is an updated version of the SENS
for Tornado manual. NPT is free and comes with some network
libraries source. There is a patch also in source on
WindSurf->Download->Patches for SPR# 30894.
5.2 TCP/IP REFERENCE BOOKS
----------------------
TCP/IP Illustrated Volume 1, by Richard Stevens
-----------------------------------------------
This book is an overview of TCP/IP protocols.
TCP/IP Illustrated Volume 2, by Richard Stevens
-----------------------------------------------
The Tornado 2.0 network stack is based on BSD4.4 Lite.
This book explains BSD4.4 code, function by function, line by
line.
UNIX Network Programming: Networking APIs: Sockets and XTI
Volume 1, Second Edition, by Richard Stevens.
----------------------------------------------------------
This book summarizes the TCP/IP Illustrated Volume 2 information
and provides lots of client/server code examples which illustrate
common programming errors.
Effective TCP/IP Programming 44 Tips to Improve Your TCP/IP
Programming by Jon C. Snader
-----------------------------------------------------------
High-Speed Networks TCP/IP and ATM Design Principles
by William Stallings.
----------------------------------------------------
This book discusses TCP/IP performance and memory resource
issues.
5.3 WindSurf->Technical Solutions II
--------------------------------
How does Tornado 2.0 (SENS) support END and BSD44 drivers at the same time
How do END driver pass link layer information to the MUX
Network Driver Porting Issues
TROUBLESHOOTING SIMPLE NETWORK PROBLEMS IN TORNADO 2.0
------------------------------------------------------
If you are new to Tornado 2.0, or you are booting a target for
the first time, or if the target booted successfully, but does
not respond to pings, please read basicNetConfiguration article.
basicNetConfiguration contains instructions on how to configure a
kernel with a target shell and network debugging libraries. It
contains information to avoid errors like:
rpccore backend client RPC: Timed out
Network interface unknown
muxDevLoad failed for device entry
wdbConfig: error configuring WDB communication interface
Error loading file: errno = 0xd0003
If target responded to pings initially, and after running your
socket application (or a library that uses the network such as
the FTP server), applications sputter or hang for a while and
then resume, please read netPerformance article. It gives an
overview of vxWorks network internals and configuration issues
that impact performance.
This article assumes that a target has booted successfully and
responds to pings initially. You are either running demo code
from WindSuf->Demo Code->Networking or your own application.
At some point, the target does not respond to pings anymore, or
network performance is slow. At this point from the target shell,
verify that tNetTask 