Wireless data radio manufacturers are not unlike data networking equipment manufacturers in that they all claim to have the fastest data transmission speeds and lowest latency. Each will boast their unique design is superior and faster than everyone else’s. But experienced networking professionals know that such performance claims are based on artificial laboratory conditions using minimally configured boxes, fed with simple, repetitive data streams. These test schemes are devised to squeeze out as many megabits per second as possible. The big question then, is how can buyers compare the actual performance of devices and get an idea of how they will truly perform in a real environment?
This subject is becoming more important because the complexity of wireless data communications devices is increasing. These boxes aren’t just passing data at the physical or radio frequency layer. They are performing tasks like switching, routing, encryption, filtering, packet inspection and more. All of these tasks require more processor cycles, storage, and resources than before. Buyers need to become educated to see beyond the manufacturer claims.
Using a Standard Framework
What is needed is a standard or framework for testing these devices that would allow them to be objectively compared. Fortunately, there is one. It is the Internet Society’s Request for Comments number 2544 (RFC 2544), “Benchmarking Methodology for Network Interconnect Devices.” Although written in 1999 for wired devices, much of it is applicable to wireless backhaul point-to-point equipment (microwave) as well as point-to-multi-point wireless data systems, including WiFi and LTE. These devices are increasingly taking on greater data networking functions such as routing and switching, and therefore, can be tested just like their wired counterparts.
Buyers specifying wireless data communications devices should ask vendors and manufacturers if they have test results available that are fully compliant or conditionally compliant with RFC 2544. This is a necessary because a specification or cut-sheet does not provide enough information to understand a device’s data networking performance.
These devices are too complex, and their performance can vary greatly based on the type of data they are passing and by how they are configured. In some instances, the enabling of one feature can cause a dramatic impact to CPU cycles when it’s implemented in software, meaning that some basic switching functions may no longer be handled by specialized integrated circuits that normally perform these functions.
Understanding RFC 2544
All buyers should familiarize themselves with the RFC 2544 standard. It’s a concise, 31-page document describing the testing methods to use, and how the test results should be presented. The guidelines are in terms of “MUST”, “SHOULD”, and “MAY”. If a test implementation does not satisfy all “MUST” guidelines then it is not compliant. Conditional compliance is the term used for implementations that satisfy all “MUST” guidelines but not all “SHOULD” guidelines.
The recommended test set up for wireless communications devices includes a tester and two wireless devices, each deemed by the RFC as a Device Under Test (DUT). This setup allows performance tests to be run. In the basic sense, testing is done by having a tester send Ethernet frames into one end using a variety of Ethernet frame sizes, frame types and speeds, and then counting and analyzing the output at the other end.
Configuration for the Device Under Test (DUT)
One way RFC 2544 prevents vendors from exaggerating performance is requiring tested devices to be configured in a manner “following the instructions provided to the user.” This one sentence is important. It prevents a vendor from disabling or turning off all but the device’s bare bones frame switching functionality, as a means of boosting throughput or reducing latency. RFC 2544 goes on to say that it is “expected that all of the tests will be run without changing the configuration or setup of the DUT in any way other than that required to do a specific test.” This is meant to prevent manufacturers from choosing an optimum configuration for each test and prevents configuration tweaking between tests for the sake of performance optimization.
The RFC describes the type of test frames to use, the addresses, and how the frame size should be varied to obtain the most accurate performance results. Be leery of test results that do not specify the frame type or size used to obtain them.
Tests Should Be Run for a Variety of Specifications
Throughput
Latency
Security filters and encryption
Unidirectional, bidirectional, multi-port and multi-stream traffic
Bursty traffic
Broadcast and management frames
Frame loss rate
System recovery
An article for Radio Resource International e-magazine (July 2015 - p. 18) provides details on each of these specific tests.
RFC 2544 was originally written for wired devices such as routers and switches, and it provides an excellent series of tests to better understand the data processing and forwarding abilities of these devices. Radio frequency (RF) matters are not included in the tests. However, it would not be unreasonable to expect that these tests could be repeated using simulated good and poor RF conditions, by increasing attenuation or introducing noise, in order to further gauge a wireless device’s ability to forward data in all environments.
RFC 2544 is the best standard the industry has toward gaining a better understanding of data processing performance of wired and wireless devices. Perhaps in the future it will be expanded to specifically include wireless devices and some variation of RF conditions will be included.
