Using a Standard Framework What is needed is a standard or framework for testing these devices that would allow them to be objectively compared. Fortunately, there is one. It is the Internet Society’s Request for Comments number 2544 (RFC 2544), “Benchmarking Methodology for Network Interconnect Devices.” Although written in 1999 for wired devices, much of it is applicable to wireless backhaul point-to-point equipment (microwave) as well as point-to-multi-point wireless data systems, including WiFi and LTE. These devices are increasingly taking on greater data networking functions such as routing and switching, and therefore, can be tested just like their wired counterparts. Buyers specifying wireless data communications devices should ask vendors and manufacturers if they have test results available that are fully compliant or conditionally compliant with RFC 2544. This is a necessary because a specification or cut-sheet does not provide enough information to understand a device’s data networking performance. These devices are too complex, and their performance can vary greatly based on the type of data they are passing and by how they are configured. In some instances, the enabling of one feature can cause a dramatic impact to CPU cycles when it’s implemented in software, meaning that some basic switching functions may no longer be handled by specialized integrated circuits that normally perform these functions. Understanding RFC 2544 All buyers should familiarize themselves with the RFC 2544 standard. It’s a concise, 31-page document describing the testing methods to use, and how the test results should be presented. The guidelines are in terms of “MUST”, “SHOULD”, and “MAY”. If a test implementation does not satisfy all “MUST” guidelines then it is not compliant. Conditional compliance is the term used for implementations that satisfy all “MUST” guidelines but not all “SHOULD” guidelines. The recommended test set up for wireless communications devices includes a tester and two wireless devices, each deemed by the RFC as a Device Under Test (DUT). This setup allows performance tests to be run. In the basic sense, testing is done by having a tester send Ethernet frames into one end using a variety of Ethernet frame sizes, frame types and speeds, and then counting and analyzing the output at the other end. Configuration for the Device Under Test (DUT) One way RFC 2544 prevents vendors from exaggerating performance is requiring tested devices to be configured in a manner “following the instructions provided to the user.” This one sentence is important. It prevents a vendor from disabling or turning off all but the device’s bare bones frame switching functionality, as a means of boosting throughput or reducing latency. RFC 2544 goes on to say that it is “expected that all of the tests will be run without changing the configuration or setup of the DUT in any way other than that required to do a specific test.” This is meant to prevent manufacturers from choosing an optimum configuration for each test and prevents configuration tweaking between tests for the sake of performance optimization. The RFC describes the type of test frames to use, the addresses, and how the frame size should be varied to obtain the most accurate performance results. Be leery of test results that do not specify the frame type or size used to obtain them. Tests Should Be Run for a Variety of Specifications Throughput Latency Security filters and encryption Unidirectional, bidirectional, multi-port and multi-stream traffic Bursty traffic Broadcast and management frames Frame loss rate System recovery An article for Radio Resource International e-magazine (July 2015 - p. 18) provides details on each of these specific tests. RFC 2544 was originally written for wired devices such as routers and switches, and it provides an excellent series of tests to better understand the data processing and forwarding abilities of these devices. Radio frequency (RF) matters are not included in the tests. However, it would not be unreasonable to expect that these tests could be repeated using simulated good and poor RF conditions, by increasing attenuation or introducing noise, in order to further gauge a wireless device’s ability to forward data in all environments. RFC 2544 is the best standard the industry has toward gaining a better understanding of data processing performance of wired and wireless devices. Perhaps in the future it will be expanded to specifically include wireless devices and some variation of RF conditions will be included. Contact us to learn more about what we can do for you.