This is a reprint of a of "Packet Burst Update: BNETX vs. VLM Implementations" in the November 1993 Issue of NetWare Application Notes. To order a copy of this application note, which includes graphics not found in this reprint, see the end of this article. NOVEMBER 1993 Packet Burst Update: BNETX vs. VLM Implementations Myron Mosbarger Senior Consultant Systems Research Department Drex Dixon Systems Software Engineer NetWare Systems Group Novell's new implementation of packet burst in the DOS Requester(VLMs) offers several advantages over traditional packet windowing implementations and over Novell's previous packet burst shell (BNETX).This AppNote compares the old and new implementations, focusing on the architectural changes that have been made. By understanding the use of PB buffers and how each version uses window size and interpacket gap for dynamic self-adjustment, network administrators will have the conceptual foundation for installing or updating packet burst on their own network. Copyright þ 1993 by Novell, Inc., Provo, Utah. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise,without express written permission from Novell, Inc. Disclaimer Novell, Inc. makes no representations or warranties with respect to the contents or use of these Application Notes (AppNotes) or of any of the third-party products discussed in the AppNotes. Novell reserves the right to revise these AppNotes and to make changes in their content at any time, without obligation to notify any person or entity of such revisions or changes. These AppNotes do not constitute an endorsement of the third-party product or products that were tested. Configuration(s) tested or described may or may not be the only available solution. Any test is not a determination of product quality or correctness, nor does it ensure compliance with any federal, state, or local requirements. Novell does not warranty products except as stated in applicable Novell product warranties or license agreements. Contents Introduction . . . . . . . . . . . . . . . . .63 Brief Review of Packet Burst. . . . . . . .63 The Old Implementation (BNETX) . . . . . .63 The New Implementation (VLM) . . . . . .64 Key Concepts . . . . . . . . . . . . . . . . .64 Buffering . . . . . . . . . . . . . . . . .64 Windowing . . . . . . . . . . . . . . . . .64 Interpacket Gap (IPG) . . . . . . . . . . .66 Comparison of Packet Burst in BNETX and VLMs .67 PB Buffers . . . . . . . . . . . . . . . .68 PB Buffers in BNETX . . . . . . . . . .68 PB Buffers in VLMs . . . . . . . . . . .70 Window Size and Interpacket Gap . . . . . .71 Window Size in BNETX . . . . . . . . . .71 Window Size in the VLMs . . . . . . . .72 Adjusting the Window Size . . . . . . .74 Tuning the Interpacket Gap . . . . . . . .75 The Optimum Delivery Rate . . . . . . .75 Determining the IPG in VLM 1.02 and Earlier . . . . . . .75 Determining the IPG in VLM 1.03 and Later . . . . . . . .76 Pings Over Satellite Links . . . . . . .77 Congestion . . . . . . . . . . . . . . . .78 How BNETX Handles Congestion . . . . . .78 How the VLMs Handle Congestion . . . . .79 Conclusion . . . . . . . . . . . . . . . . .79 Trademarks Novell, the N design, and NetWare are registered trademarks, and Internetwork Packet Exchange, NetWare Loadable Module, NLM, NetWare DOS Requester, NDR, Packet Burst, Virtual Loadable Module, and VLM, are trademarks of Novell, Inc. All other product names mentioned are trademarks of their respective companies or distributors. Introduction Two major concerns as local area networks have grown into metropolitan and wide area networks (WANs) are bandwidth utilization and response time. Novell developed packet burst technology as an enhancement to its NetWare Core Protocol (NCP), primarily to increase available bandwidth and reduce response time on busy network segments and over low-speed links. The first implementation of packet burst consisted of an add-on module (PBURST.NLM) for the NetWare 3.11 operating system and a burst mode version of the NETX shell (BNETX.EXE). With the release of NetWare 4.x and 3.12, packet burst has been redesigned and integrated as a feature in the core products. The workstation packet burst capabilities are integrated into the new VLMs (Virtual Loadable Modules) and are enabled in the default workstation configuration. The differences between BNETX and the VLM implementations of packet burst are significant. This AppNote focuses on the changes in Novell's packet burst implementation and identifies the effects they have on network throughput. Brief Review of Packet Burst NetWare's original service protocol, NCP, was designed and optimized for use over high-speed LAN links (2.5 Mbps and faster). The one-to-one request and response ratio imposed by NCP did not present a problem for local area communication. As NetWare users began communicating over slower wide area links, some of the limitations of this request/response dialogue were accentuated. The Old Implementation (BNETX) To improve throughput during large file transfers, Novell's initial implementation of packet burst modified the number of packets sent in response to a request. This allowed a many-to-one ratio rather than a one-to-one ratio when a client was reading or writing large files (over 64KB). The process was dynamic and improved throughput during file transfer by increasing the number of packets that could be sent before receiving an acknowledgement. This required modifications to the workstation shell (NETX), which culminated in the creation of a new shell called BNETX and a new NetWare Loadable Module (NLM) called PBURST.NLM. Generally, users on wide area networks (typically slower links) stood to gain the most from using BNETX. Implementing an optimal configuration with BNETX required manual modification of the workstation's NET.CFG file. For in depth information regarding Novell's packet burst shell, see "An Introduction to Burst Mode Technology" in the March 1992 NetWare Application Notes. The New Implementation (VLM) While the first iteration of packet burst was successful in many installations, its ability to dynamically configure and adjust to network conditions was limited. In the current implementation of Novell's DOS Requester, packet burst is automatically enabled at the workstation in the FIO Virtual Loadable Module (VLM). At the server, packet burst is an integral component of the NetWare 4.x operating system. No special code is required. NetWare 3.12 also has packet burst capabilities integrated into the operating system. Note: If workstations will be running the new VLMs and connecting to file servers running NetWare 3.11, the PBURST.NLM must be installed on those servers to take advantage of the new VLM packet burst features. Key Concepts Before proceeding with our comparison of the BNETX and VLM packet burst implementations, it is helpful to have some basic understanding of buffering, windowing, and interpacket gap. These are key to understanding the major architectural differences between traditional windowing protocols, BNETX, and the VLMs. Buffering To understand how Novell's packet burst VLM works, we must first understand where and how buffering is used in NetWare communications. There are two kinds of buffers which affect throughput:  The network interface card hardware buffers, which are responsible for taking packets off the network and passing them to the receiving device.  Workstation (shell) and router buffers, which are responsible for copying or transferring packets, or their contents, to memory. As we will see, hardware and software buffers can significantly limit the amount of data a workstation or router can handle. Windowing Traditional windowing implementations focus on sending groups of packets back-to-back in what is called a "window." Once all of the packets in the window have arrived at the destination, a single acknowledgment packet is returned (see Figure 1). Figure 1: In traditional windowing protocols, a group of packets is sent with a single acknowledgement packet. Provisions are made in case not all of the packets in the window arrive intact. For example, if 10 packets were requested, the receiving station tracks the packets it receives by sequence number. If any packets are not received within the timeout period, the receiving station requests are-send. The re-send begins with the lost packet (packet #5 in the example shown in Figure 2) and includes all packets in sequence to the end. Figure 2: When packets are dropped, the packets are re-sent starting with the one that was bad. In traditional windowing implementations, adjusting the window size is the primary means of tuning throughput. This tuning process is illustrated in Figure 3. Figure 3: When packets are dropped, the window size is reduced in traditional windowing implementations. A small number of dropped or missing packets is to be expected due to varying network traffic conditions. But when a specified number of packets is dropped in so many windows in a row, the window size is adjusted downward. Decreasing the window size reduces the probability of dropped packets, which generally makes the file transfer process more efficient because there aren't as many re-sends. However, as the window size becomes smaller and smaller it can also reduce the effectiveness of the windowing protocol. After all, the whole point is to be able to send a group of packets instead of a single packet for each acknowledgement. Interpacket Gap (IPG) Interpacket gap is the time that elapses between the end of one packet and the beginning of the next, as illustrated in Figure 4. Adjusting the IPG (or packet metering, as it is sometimes called) is a technique not commonly used in traditional windowing protocols. As explained above, traditional windowing implementations focus on adjusting the window size as the sole means of managing throughput. Novell's latest packet burst implementation adjusts the IPG as the primary variable for increasing throughput, then adjusts the window size as the secondary variable. Figure 4: The interpacket gap is the time between the end of one packet in the window and the beginning of the next. Adjusting the IPG is a technique used in Novell's VLM implementation of packet burst. It provides greater throughput than adjusting the window size because the number of packets per window remains the same. By adjusting the IPG, packets flow through the workstation in larger windows and with no retries. Figure 5: Adjusting the IPG controls the flow of packets to match the ability of the receiving NIC to handle them. Optimally, the packet receive rate is equal to the workstation's capacity to receive and distribute the data with no dropped packets. The VLMs dynamically adjust and optimize the IPG to overcome the limitations of hardware processing speed and hardware buffering, as shown in Figure 5. Comparison of Packet Burst in BNETX and VLMs There are three major differences between packet burst in BNETX and in the VLMs. They are:  Their use of workstation buffers (PB Buffers)  Their use of window size and interpacket gap as tuning variables  Their ability to self-configure and handle congestion PB Buffers PB Buffers in BNETX. To configure the number of buffers the shell can use for packet burst, workstations running BNETX require a modification to the NET.CFG file, similar to the following: PB Buffers = 4 (or some other decimal value) This value determines the amount of memory that will be claimed by the shell for packet burst buffers. The formula for calculating the memory requirement associated with BNETX is: Memory required by shell = PB Buffers X Packet Size For example, if the packet size is 1500 bytes (as it is for Ethernet), the amount of additional memory allocated to the workstation shell to run BNETX with four buffers is 6000 bytes (4 X 1500 = 6000). For Token Ring (with 4096-byte packets), the same configuration would require 16,384 bytes of memory. Note that this is non-returnable memory. Once it has been pre-allocated to the shell for packet burst buffers, the memory is no longer available for any other use. A number of other factors come into play here. For one, the NetWare shell has a 64KB segment limit, which makes it possible to specify a PB Buffers value greater than the shell can accommodate. For example, if the current shell size is 43KB, the maximum available memory left for buffers would be 21KB (64KB X 43KB). As shown in Figure 6, this much memory could accommodate fourteen 1500-byte Ethernet buffers or five 4096-byte Token Ring packets. Figure 6: The 64KB segment limit of the BNETX shell limits the amount of buffer space available. With BNETX, performance generally improves as the number of buffers increases. However, there is a point after which adding more buffers yields only nominal performance benefits. For example, experience has shown that a PB Buffer value of 10 for 4096-byte Token Ring packets provides no more performance benefit than a value of 5 would. Another weakness with BNETX is its inability to dynamically configure the number of buffers needed. Arriving at an "optimized" configuration requires user intervention on each workstation using packet burst. To discover the optimal number of buffers, a user must experiment. Without the aid of network analyzers, this is mostly a trial-and-error process. Once the proper number of buffers is determined, you must examine the workstation to see if it has enough memory to support the packet burst configuration along with the other applications running at the workstation. In addition to its stringent memory requirements, the way BNETX uses buffering requires two moves or copies to transfer data into workstation memory (see Figure 7). Figure 7: BNETX requires two separate transfers or copies to get data from the NIC memory into workstation memory. The latency factor as packets are first transferred into buffer memory and then into workstation memory limits the overall speed at which packets could be received. The net result is a decrease in actual throughput. PB Buffers in VLMs. In Novell's VLM implementation, the workstation packet burst feature is contained in the FIO.VLM module. (FIO stands for File Input/Output.) Like BNETX, FIO.VLM uses the "PB Buffers = " statement in NET.CFG, which is set by default to a value of 3. Unlike BNETX, the VLM does not use this value to set aside buffer space. Rather, the value either enables (non-zero integer) or disables (zero) the packet burst feature at the workstation. For example, if packet burst feature will not be used, a small amount of memory can be reclaimed by adding the following command to the NET.CFG file: PB Buffers = 0 This command disables packet burst at that workstation and reduces the DOS Requester's memory requirement slightly. Another difference is that the FIO.VLM does not buffer read packets into an intermediate buffer pool as BNETX did. It requires only one copy to transfer memory from the NIC to the workstation (see Figure 8). Figure 8: The VLM implementation requires only one transfer to get data from NIC memory into workstation memory. This major architectural change provides several benefits over BNETX. The most significant advantage is the VLM's ability to dynamically configure buffering at each workstation -- no user intervention or NET.CFG modification is required. In addition, the VLM dynamically tunes itself for optimal performance and improves throughput by eliminating the latency effects associated with the addition of intermediate buffers. Window Size and Interpacket Gap Window Size in BNETX. BNETX is basically a traditional implementation of a windowing protocol over IPX. It adjusts the window size as its primary means of flow control. As shown in Figure 9, packets are sent in windows of a predetermined size. If packets are dropped, the window size is adjusted downward to accommodate either network traffic or the destination device's capacity to receive packets. With BNETX, the IPG remains relatively constant and does not generally improve performance. Figure 9: The BNETX implementation adjusts the window size as its primary means of flow control. Determining the appropriate window size is crucial to the tuning process. In Novell's BNETX implementation, the initial window size is determined by the shell, the PB Buffers parameter, and the packet size--with a maximum window size of 64KB. BNETX requires the number of workstation buffers to be set by editing the NET.CFG file. The workstation then configures its shell memory buffers when NET.CFG is run, as described above. For that reason, BNETX will never request a window size larger than its buffering capacity. It is important to understand that the shell or memory buffering capacity of the receiving device is independent from the hardware buffering capabilities of the network interface card. Therefore, setting the buffer size higher than the NIC can handle will in effect waste memory. In BNETX, the maximum buffering capabilities of a workstation is the weakest buffering component. Window Size in the VLMs. Version 1.02 and earlier versions of the VLMs do not adjust the window size, only the IPG. VLMs version 1.03 and later do adjust the window size, but only after adjusting the IPG to its maximum value first. For VLM versions 1.02 and later, window sizes are determined as follows. By design, the VLM implementation has no window size limitation as BNETX does (this has been successfully tested and confirmed). However, to accommodate the capabilities of the majority of network devices, the values of 16 packets for a read and 10 packets for a write have been set as defaults. The maximum window size is calculated as follows: Maximum number of reads (16) X the frame size Maximum number of writes (10) X the frame size To illustrate how these imposed defaults work, suppose a workstation attached locally on a network segment wants to read a 56KB file. On Ethernet, the workstation will request two windows (16 packets) of 24KB and one window (6 packets) of 8KB. On Token Ring with 4KB packets,the workstation would have one window (14 packets) of 56KB (see Figure 10). Figure 10: Imposed defaults for maximum window size determine how many packets can be sent in a burst. Window sizes of 16 for reads and 10 for writes were selected for two reasons. First, they provide a greater reliability in a wide variety of installations. On wide area networks or slower links (which are generally more prone to loss than LAN links), larger window sizes tend to be less reliable. Second, as the number of packets in a transfer increases, the percentage of improvement increases substantially within the first 10 to16 packets. Beyond that, performance improvements are minimal (see Figure 11). Figure 11: As the size of the burst increases, so does the percentage of performance improvement. Starting with VLM version 1.02, two new configuration parameters have been added to allow users to modify the packet burst window size: PBURST READ WINDOW SIZE = n (3 to 255) PBURST WRITE WINDOW SIZE = n (3 to 255) When entered in the NET.CFG file, these commands allow a user to override the defaults. For users on a locally-attached segment whose network and server have sufficient capacity, increasing the number of packets in a read or write will improve performance even more. Adjusting the Window Size. The window size is adjusted exponentially. As an example, suppose a window size of 16 packets with a maximum packet size of 1KB reaches it maximum IPG and the percentage of unsuccessful bursts is unacceptable. The window size will be decreased according to the following formula: NewWinSz = CurWinSz þ ((CurWinSz þ 3 þ MaxPcktSz) / 4) In our example, the new window size would become 12KB. The calculation is as follows: 16KB þ ((16KB þ 3 þ 1KB) / 4) = 12KB If a 12KB window size was also found to be unacceptable, the size would be reduced to 10KB, and so on. Tuning the Interpacket Gap Unlike traditional windowing implementations, the VLMs substantially improve throughput by "tuning" the IPG. This technique is unique to Novell's VLM packet burst implementation. The Optimum Delivery Rate. Conceptually, the optimal window delivery rate sends data at the same rate the slowest device along the packet delivery path can process it, without dropping any packets. This would create a smooth flow of data with the least amount of overhead (timeouts, re-sends, and so on). Novell has found that by adjusting the interpacket gap, very large windows of data can be sent successfully. Determining the appropriate interpacket gap is the key to this process. Ideally, when the gap is optimally tuned, a workstation can read or write a window of any size. Determining the IPG in VLM 1.02 and Earlier. Since VLM version 1.02 adjusts only the IPG, not the window size, it is important to determine an appropriate maximum IPG or ceiling for a window. To do so, the workstation sends a series of pings (messages that go to the destination and back again). Half the time value of fastest round trip becomes the maximum interpacket gap (see Figure 12). Figure 12: To arrive at the maximum IPG, the workstation takes half of the fastest round-trip ping time. With this maximum value established, the requesting device begins the file transfer by using an IPG of 0 (zero). When a predetermined number of failures occur (for example, two packets dropped in six bursts), the interpacket gap is automatically increased until either no packets are dropped or the maximum IPG is reached. As an example, suppose a workstation determined a maximum IPG value of 5000. The initial IPG is set to zero. If this is unsuccessful, the IPG is increased to 3000. If this is successful, it remains unchanged. This process is illustrated in Figure 13. Figure 13: The process of adjusting the IPG in VLM versions 1.02 and earlier. The ceiling value is large enough to handle nearly all network and workstation bottlenecks. Since the VLMs pass data directly to memory on the workstation, the only buffer limitations are those of the interface card (or in the case of a router, both hardware and receive buffers). Remember, for VLMs version 1.02 and earlier, only the IPG is changed, not the window size. Determining the IPG in VLM 1.03 and Later. In the latest version of the VLMs, the initial IPG is set to half of the maximum IPG and uses a binary algorithm to reach the optimum IPG. This represents two changes:  First, by setting the initial IPG at half the maximum rather than zero, the number of re-sends is reduced for the majority of today's workstations.  Second, the binary algorithm for tuning the IPG is substantially faster than the method used in VLM 1.02, and allows for fine tuning the gap time to maximize performance. These two changes can improve login time for packet burst-enabled workstations. Figure 14 illustrates how the IPG tuning method works for VLM versions 1.03 and later. Figure 14: The IPG tuning method for VLM versions 1.03 and later. In this example, if the maximum IPG is 5000 microseconds, the initial IPG would be set at 2500. If this were successful, the IPG would be decreased to 1300 microseconds. If that were successful, the IPG would be reduced to 700 microseconds. If that were unsuccessful, the IPG would be increased to 1000 microseconds. This process would continue until a baseline IPG is established. Note: When decreasing the IPG, the algorithm rounds up (1250 becomes 1300). When increasing the IPG, the algorithm rounds down (1250 becomes 1200). The method used in version 1.03 will be faster for the majority of client workstations and will tune all machines faster than the 1.02 method. Pings Over Satellite Links. When dealing with satellite or network links with inherent delays, the original ping process produced an unrealistic value. To accommodate this type of link, the ping process was modified to determine an approximate available bandwidth. This is accomplished by sending three pairs of pings across the network link. The time between the end of the first packet and the end of the second packet in each pair is computed and saved. The average of all three pings is computed and becomes the maximum IPG for the workstation. Based on this value, the IPG is set to half the value for the first transfer and is then tuned for optimal throughput (see Figure 15). Figure 15: Modified process for determining IPG over satellite links with inherent delays. This method provides a good approximation of the available bandwidth across the route. Congestion A potential problem with high volume, "bursty" traffic is bandwidth availability and congestion. Traditional packet burst implementations can add additional traffic. For example, if a burst of packets begins transferring and packets are dropped (due to increased traffic) during the transfer, retry algorithms tend to increase congestion with retry packets. This creates additional traffic on an already busy network. The effect of this problem as seen by the users could be lost network connections,slow response, or other related timeout problems. When packets are dropped, it is the function of the client or workstation software to initiate retries. Client software that is unable to dynamically adjust or back off the number of packets and retries in high traffic conditions tends to worsen the problem by increasing the number of packets sent, which was the cause of the problem initially. How BNETX Handles Congestion. BNETX adjusts to network congestion in the traditional manner by adjusting the window size. The window size is temporarily reduced and will eventually return to the original baseline size. The smaller the window, the more windows are needed to complete a transfer and with each window an acknowledge packet is sent. This means that packet burst becomes less effective in high traffic situations. How the VLMs Handle Congestion. The VLM deals with congestion by increasing the IPG. The VLM checks for network congestion by monitoring and tracking the number packets lost during a transfer. For example, if one packet is lost in six windows, the VLM makes no corrections. However, if a packet is dropped in consecutive windows, the VLM would increase the IPG for the next several windows. This process will continue until the number of packets dropped is within an acceptable limit (see Figure 16). Figure 16: The VLM handles network congestion by increasing the IPG. Remember, the retry timeout and IPG values are based on information received about the network's capacity prior to the file transfer. The primary difference between adjusting the window size and adjusting the IPG is that the number of acknowledgements does not increase as traffic increases. This means that when windows of 24KB are being sent, there is no loss in the efficiency of the packet burst protocol even though the packets arrive at a slower rate. In contrast, there is a loss of efficiency when adjusting the window size. Conclusion In NetWare 3.12 and 4.x, the FIO.VLM contains Novell's new implementation of packet burst for workstations. This implementation offers several advantages over typical packet burst implementations and over Novell's previous version (BNETX). While VLM versions 1.03 and later have been significantly improved for locally-attached users, they have been modified specifically to provide increased performance for low-speed links and links with delays over VSAT (Very Small Aperture Terminal). The percentages shown in Figure 17 are representative of the performance benefits that come from using BNETX or the VLMs over NETX. Keep in mind that network interface cards, drivers, workstations,and servers all influence total throughput. Our testing was mainly conducted to show the effect of the tuning algorithms for the VLMs compared to BNETX. Figure 17: Percentage of performance increase for BNETX and the VLMs as compared to the NETX shell. Sites which are doing large file transfers or which are currently using BNETX will benefit substantially by installing version 1.03 of the VLMs. About Novell Research Novell Research is a program through which Novell publishes technical information about designing, implementing, and managing NetWare-based systems. Publications produced by Novell Research include NetWare Application Notes and Novell Research Reports. Novell Research publications are written by Novell consultants and engineers, with the help of outside industry experts when necessary. The information contained in these publications is based on lab research and actual field experience of the authors, covering topics in these main areas: o Network design and optimization strategies o Network management tactics o NetWare internals and theory of operations o Novell product implementation guidelines o Integration solutions for third-party products o NetWare programming techniques This information is designed to benefit Novell's technical audience consisting of systems engineers, support engineers, consultants, programmers, network managers, and information systems personnel. Novell's Systems Research Department in Provo, Utah, serves as home to Novell Research. However, the name "Novell Research" and the program represent all of Novell's geographic locations and business units. NetWare Application Notes NetWare Application Notes (AppNotes) is published monthly by Novell. Each issue contains several articles that address different aspects of working with NetWare, including design, implementation, development, and optimization. The material is based on technical research performed by Novell personnel. Novell Research Reports Novell Research Reports are in-depth treatments of key technical issues such as network backup and security. These reports also represent technical research performed by Novell personnel. Cooperative research projects are planned for topics of general interest for which Novell cannot produce strategic research without outside assistance. Subscription Rates NetWare Applications Notes are available directly from Novell, by subscription only. As of March 1, 1993, the one-year subscription rate for domestic orders is $95. The one-year subscription rate for international orders is US$135. AppNote subscribers also receive copies of selected Novell Research Reports as they become available. A discount is available for organizations desiring 10 or more subscriptions. Back issues of NetWare Application Notes are available for $15 each, and back issues of Novell Research Reports and Cooperative Research Reports are $25 each (add $5 for shipment outside the U.S.). A discount is available for bulk reprint orders of 10 or more copies. Contact the Novell Research Order Desk for more information. All orders must be prepaid. Novell Research Order Desk: Voice (toll-free) 800-377-4136 Voice 303-297-2725 Fax 303-294-0930 A list of available Novell Research publications follows. Novell Research Publications List To obtain an updated list of publications, call the Novell Research Order Desk at 800-377-4136. Outside the U.S. and Canada, call 303-297-2725. To order NetWare Application Notes (AppNotes) subscriptions, AppNote reprints, or Novell Research Reports, Call toll-free (800) 377-4136. Outside the U.S., call (303) 297-2725. Please have your order and credit card information ready. For the November 1993 appnote, specify part number 164-000032-011.