On-the-fly capture and replay mechanisms for multi-port network devices in operational networks

Testing network devices in a live environment is desirable due to its reality. However, the defects are not reproducible, and the network connectivity will be broken if the device is down. For effective defect reproduction from real traffic, we design a new mechanism, which allows the device under test (DUT) to be automatically online/offline, and supports multi-port replay for multi-port network devices with an OpenFlow switch. The defect traces are captured when the DUT is online. When a DUT failure is detected, the DUT will be offline, and the defect-triggering traces will be replayed to identify the defect. For efficient replay, we keep only partial payloads in a reduced number of packets in the defect traces that are sufficient to trigger the defects. For defect identification, reduction based on a binary search algorithm is presented to deal with the defects caused by payload anomalies and by overloading. The downsizing ratios in the cases of payload anomalies and overloading are up to 98.8% and 96%, respectively. The minimum outage time of the failover during the DUT failure is obtained when the check interval is 1 second and the number of tolerable consecutive failures is 2.