The speeds of Ethernet communication links used in the datacenter continue to increase and next generation network adapters based on 100GbE and 400GbE Ethernet (802.3xx) standards are already in active product development and testing. Although these links employ fiber, signals must still travel over copper backplanes and interconnects.
Achieving these speeds over copper requires the use of equalization (pre-emphasis and de-emphasis) of signals to compensate for transmission losses and introduces several new PHY-layer and protocol debug challenges. One of the more daunting of these challenges is debugging link training sequences, a requirement first seen in the 40GBASE-KR4 specification and likely to be included in many more 802.3xx standards going forward.
Link training is needed as speeds increase because Ethernet is an open-channel protocol. Whether the channel is a backplane or a cable, the device does not know in advance what the channel characteristics will be. Link training is what allows devices to optimize their equalization to the connected channel before network traffic starts to flow. Link training is complex and takes a long time relative to high-speed data rates – up to 500 ms. This is close to an eternity when an engineer is analyzing individual bits on a 25 Gb/s link.
To ensure the reliable operation data center managers expect, IEEE802.3 modules that involve link training need to be tested with each other, which often reveals link training bugs. But tracking down those bugs is extremely difficult and time consuming. If link training lasts 500 ms, there could be up to 1 million control channels -- the lower speed part of the signal with the protocol data – each with potential errors. Trying to decode that many channels by hand is impractical at best, so until now debug has involved lots of trial and error. Extensive experience is helpful too.
With our customers facing a potentially show-stopping debug problem, Tektronix has stepped up to tackle the complexity of KR link training debug. The result of this effort is the recently introduced Option HSSLTA (High Speed Serial Link Training Analysis) tool for use with our high-performance DPO70000SX series real-time oscilloscope.
It’s safe to say that the introduction of the industry’s first link training debug and analysis tool has been well received. First, EDN’s Martin Rowe posted a detailed article about the new tool and more recently we learned that it has been named a finalist in the 2017 Golden Mousetrap Awards. The winners will be announced Feb. 7, 2017 in Anaheim, Calif. during MD&M West.
The Option HSSTLA tool gathers PHY-level and link-level data on the negotiation process. It then works through millions of frames to isolate those where equalization changes take place. From the captured and time-stamped frames, the tool can analyze the data or it can be downloaded for analysis and storage.
The figure below shows this process in action. In this example, the software has taken negotiation data from a 25 Gb/s signal and placed markers at the top to indicate frames where negotiation takes place. Each frame (marked "FM") contains four hex words that define the negotiation data. The table in the screen image displays state changes in the process, including columns that indicate equalization coefficients
Option HSSLTA captures and triggers on link training data.
Even with Option HSSLTA, debugging modern high-speed links will never be simple. The difference now, at least for engineers working on link training design and debug, is that they now have some very relevant data on their side and insights they never had before. From a big picture perspective, Option HSSLTA will play an important role in enabling Tektronix customers keep up with the demand for 100GbE and 400GbE networking technologies.